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SECTION  I 
INTRODUCTION 

The  Department  of  Defense  (DOD)  Budget  Guidance  Manual  (Change  5,  12  August 
1977)  promulgated  a change  in  budgeting  practice  for  first  item  training  devices 
which  required  that  certain  of  these  devices  be  procured  using  Research,  Develop- 
ment, Test,  and  Evaluation  (RDT&E)  funds.  Specific  guidance  was  contained  in 
Director  of  Budget  and  Reports  memorandum  dated  14  December  1977  as  follows: 

Training  Devices.  Training  devices  that  employ  new  or 
off-the-shelf  computers  and  system  components,  but  have  train- 
ing system  unique  software  and  interface  components,  will  be 
developed  and  procured  with  RDT&E  funds.  Typically,  these 
training  devices  have  small  quantity  requirements  and  the 
initial  or  prototype  system  is  used  for  operational  training. 

The  initial  or  prototype  system,  and  all  of  its  support 
cost  through  service  acceptance,  will  be  funded  in  RDT&E. 

RDT&E  will  not  fund  beyond  the  initial  system  unless  more  than 
one  full  system  is  required  to  demonstrate  the  training  device 
performance. 

Termination  of  RDT&E  funding  would  occur  at  service  acceptance  of  the 
training  device(s).  A budgeting  process  consistent  with  this  policy  will  be 
implemented  beginning  FY  1980. 

DOD  and  Secretary  of  the  Navy  (SECNAV)  instructions  stipulate  that 
all  acquisition  programs  funded  with  RDT&E  monies  will  undergo  a test  and 
evaluation  (T&E)  cycle  as  part  of  the  acquisition  process.  The  completion  of 
various  phases  of  T&E  demonstrates,  in  part,  the  achievement  of  those  program 
objectives  and  milestones  that  serve  as  the  pacing  function  of  individual  acqui- 
sition programs.  No  authority  to  procure,  other  than  the  initial  or  prototype 
article,  will  be  granted  until  the  system  has  satisfactorily  completed  initial 
T&E. 


Historically,  it  has  been  the  rule  that  the  training  device  test  and 
evaluation  process  is  less  formally  structured  than  its  operational  hardware 
counterpart.  Further,  such  T&E  is  normally  oriented  toward  the  satisfaction  of 
technical  criteria,  and  evaluation  has  typically  been  subsumed  under  the  Develop- 
ment Agent  (DA)  rather  than  being  fully  independent.  In  view  of  the  movement  of 
prototype  training  devices  to  RDT&E  funding,  and  the  implications  of  this 
initiative  for  test  and  evaluation,  the  Chief  of  Naval  Operations  (CNO)  by 
memorandum  Serial  991B/644005  of  10  January  1978  suggested  that  the  Chief  of 
Naval  Education  and  Training  (CNET)  give  consideration  to  developing  the  capabil- 
ity for  independent  evaluation  of  training  devices. 

OPNAVINST  3960.10  provides  a comprehensive  and  proven  method  of  conducting 
T&E  for  operational  hardware  and  systems.  The  basic  concepts  and  policies  are 
understood  throughout  the  Naval  T&E  community.  For  this  reason,  any  training 
device  T&E  Instruction  should  be  based  on  existing  concepts  and  policies. 


5 


i 

*1 

; 

i 


n 


However,  it  was  recognized  that  training  device  acquisition  programs  differ 
sufficiently  from  operational  equipment  acquisition  programs  that  direct 
application  of  OPNAVINST  3960.10  is  not  feasible.  Thus,  an  instruction  for  the 
T4E  of  training  devices  will  differ  in  some  respects  from  OPNAVINST  3960.10,  but 
will  be  based  on  the  philosophy  underlying  present  T&E  efforts. 

PURPOSE 

The  Training  Analysis  and  Evaluation  Group  (TAEG)  was  tasked  by  CNET  to 
conduct  a general  review  of  the  subject  of  training  device  development  and 
acceptance  for  service  use.  The  purpose  of  this  study  is  to  produce  a draft 
OPNAV  instruction  which  will  provide  for  the  efficient  and  effective  acquisition, 
evaluation,  and  acceptance  of  training  devices  acquired  with  RDT&E  funds.  The 
specific  objectives  are: 

• identify  current  factors  in  all  services  impinging  on  training 
device  acquisition,  evaluation,  and  service  acceptance 

• develop  a classification  system  for  training  devices  useful  for 
evaluation  purposes 

• prepare  a proposed  OPNAV  instruction 

• coordinate  and  revise  the  proposed  OPNAV  instruction  at  a con- 
ference convened  for  that  purpose. 

ORGANIZATION  OF  THE  REPORT 

In  addition  to  this  introduction,  the  report  includes  four  sections  and  two 
appendices. 

Section  II  is  a summary  of  current  test  and  evaluation  policy  and  practices 
in  the  Navy,  Marine  Corps,  Army,  and  Air  Force.  It  provides  a foundation  upon 
which  the  proposed  instruction  is  based. 

Section  III  contains  the  technical  approach  used  and  the  guidelines  followed 
in  developing  the  proposed  instruction.  Section  IV  discusses  the  major  factors 
which  influenced  the  proposed  instruction.  Included  are  policy  factors,  management 
factors,  POM/budget  process  factors,  and  the  integration  of  the  device  acquisi- 
tion process  into  the  overall  system. 

Section  V contains  a series  of  policy  issues  for  CNET  consideration. 

Appendix  A is  the  proposed  OPNAV  instruction;  a glossary  of  terms  used  In 
this  report  is  provided  in  Appendix  B. 
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SECTION  II 


CURRENT  T&E  PRACTICES  IN  THE  MILITARY 

As  a prelude  to  the  development  of  the  proposed  OPNAV  instruction,  a summary 
of  existing  policy  and  practices  in  the  four  services  is  presented.  An  under- 
standing of  current  T&E  practices  is  necessary  since  the  proposed  procedures  are 
based  on  these  practices. 


PRACTICE.  Current  policy  directives  of  the  Navy  do  not  address  training  devices 
as  distinct  from  other  acquisitions.  However,  practice  in  the  service  differs 
from  policy  in  that  training  device  acquisition  programs  are  not  subjected  to  a 
complete  test  and  evaluation  program.  Rather,  they  are  acquired  more  by  a 
traditional  exception  to  existing  policy.  This  traditional  approach  is  compre- 
hensible when  training  devices  are  compared  to  operational  hardware  and  systems 
in  terms  of  cost,  requirements,  and  timing. 

Although  neither  the  cost  of  acquisition  nor  the  type  of  requirement  that 
led  to  the  acquisition  affect  the  actual  T&E  of  training  devices,  both  do  have 
an  appreciable  bearing  on  the  method  whereby  T&E  is  conducted.  Cost  defines  the 
level  and  who  conducts  the  Operational  Test  and  Evaluation  (GTE);  requirements 
define  the  participants. 

COST.  The  Acquisition  Categories  (ACATs)  are  established  in  OPNAVINST  3960.10 
primarily  on  the  basis  of  the  total  cost  of  an  equipment  or  system  acquisition 
program.  Major  training  devices;  i.e.,  high  cost  devices,  are  acquired  either 
in  conjunction  with  the  hardware  or  independent  of  the  hardware.  Each  type  of 
major  acquisition  is  discussed  in  the  following  paragraphs.  Minor  training 
devices  generally  fall  under  the  ACAT  IV  provisions.  This  means  no  independent 
operational  tests  and  evaluations  are  required,  and  the  DA  is  responsible  for 
all  T&E  as  well  as  the  program  management. 

Major  training  devices  acquired  in  conjunction  with  major  hardware  acquisi- 
tion programs  are  frequently  included  as  an  integral  part  of  the  hardware 
contract.  Many  elements  of  device  cost  are  subsumed  under  the  hardware  costs 
and  are  not  readily  identifiable  with  the  device.  In  addition,  the  hardware 
acquisition  manager  is,  simultaneously,  the  training  device  program  manager. 

Policy  dictates  that  the  cost  of  the  total  program  identifies  the  level  of  T&E 
required.  Although  the  hardware,  because  of  program  cost,  is  subjected  to  a 
complete  T&E  cycle,  including  an  independent  OTE,  the  training  device  is  subjected 
solely  to  acceptance  tests  which  insure  it  meets  the  technical  specifications. 

It  can  be  reasoned  that  this  is  adequate  for  the  device  since  its  specifications 
are  derived  from  the  hardware  and  the  device  is,  in  reality,  a component  of  the 
training  subsystem  of  the  operational  system.  Thus,  the  cost  of  the  device  is  a 
portion  of  the  cost  of  a subsystem  within  the  total  acquisition  cost  and, 
therefore,  not  independent  of  the  hardware  cost.  The  fallacy  of  the  rationale 
is  that  the  applicable  elements  of  educational  philosopny  are  not  considered  in 
an  OTE  derived  solely  from  technical  specifications. 

Independent  major  training  device  acquisitions  may,  or  may  not,  be  related 
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to  <i  hardware  ai.qu  i 1 1 ion  program.  Irrespective,  funding  ib  channeled  through 
the  device  DA . Despite  the  independence  of  these  buys,  and  regardless  of  ACAT, 
no  independent  OTE  is  conducted.  Service  acceptance,  which  in  this  case  means 
insertion  in  the  Navy  inventory,  is  predicated  solely  on  a technical  evaluation 
and  Fleet  Project  Team  acceptance.  Fleet  Project  Teams  are  subject  matter 
experts. 

REQUIREMENTS.  The  requirement  for  a training  device  should  always  be  established 
on  the  basis  of  a perceived  or  actual  operational  need.  These  needs  may  be 
either  direct  or  indirect.  A direct  requirement  stems  from  an  operational  need 
and  supports  the  training  system  for  new  or  modified  hardware.  An  indirect 
requirement  arises  from  a training  need  and  supports  a training  system  which 
needs  the  device  to  illustrate  a concept  or  provide  some  measure  of  hands-on 
training. 

Direct  Requirements.  Direct  training  device  requirements  are  usually  established 
by  operational  or  hardware  development  personnel.  The  need  for  training  becomes 
apparent,  and  these  people  take  the  approach  that  teaching  on  the  equipment  or 
on  a high  fidelity  simulator  will  accomplish  the  training  mission.  Generally 
direct  training  requirements  are  not  recognized,  or  acknowledged,  until  rela- 
tively late  in  the  acquisition  cycle,  or  until  the  hardware  has  become  operational. 
This  causes  time  to  become  a constraining  element.  As  a consequence,  the  training 
device  is  acquired  without  the  prior  establishment  of  a course  of  instruction 
including  training  strategies  and  performance  evaluation.  Thus,  training  personnel 
find  it  necessary  to  devise  a course  tailored  to  the  device  capabilities  rather 
than  to  design  a course  to  meet  the  specific  training  requirement.  This  sequence 
is  the  reverse  of  that  which  should  occur.  The  device  dictates  the  training 
system  requirements  rather  than  the  training  system  objectives  dictating  the 
need  for  a device. 

Indirect  Requirements.  Indirect  training  device  requirements  generally  originate 
within  the  training  conmunity.  This  results  in  the  identification  of  a requirement 
for  a training  device  to  support  an  existing  or  proposed  course  of  instruction. 

In  the  instance  where  a course  of  instruction  is  supported  by  a device,  and 
this  training  device  requires  modification  or  replacement,  a separate  problem 
arises.  Although  the  requirement  is  identified,  it  usually  is  given  a low 
priority  because  the  school  has  something  with  which  to  teach.  This  situation 
can  cause  T&E  difficulties  because  advance  planning  for  budgetary  support  and 
RDT&E  Support  for  T&E  is  nebulous  with  respect  to  time.  In  addition,  the  Training 
Command  is  not  a major  claimant  for  R&D  funds;  therefore,  the  acquisition  of 
required  funding  is  difficult  and  must  compete  with  nontraining  R&D  requirements. 

TIMING.  Most  simple  training  devices  require  approximately  18  months  from  the 
statement  of  the  requirement  to  final  government  acceptance.  Such  a brief  cycle 
permits  only  6 months  for  a contractor  to  design  and  produce  the  device;  the 
remainder  of  the  18  months  is  used  for  administrative  and  technical  planning 
functions.  As  devices  become  more  complex,  this  overall  time  is  extended.  For 
example,  any  sophisticated,  computer  controlled  simulator  which  combines  a 
visual  and  electronic  capability  will  generally  require  from  42  to  48  months 
from  the  statement  of  the  requirement  to  government  acceptance.  The  practice 
has  been  to  leave  direct  training  requirements  to  the  latter  portion  of  the 
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acquisition  cycle  in  order  that  the  greatest  amount  ot  information  on  the  hardware 
can  be  obtained.  In  terms  of  training  devices,  this  practice  can  cause  the 
program  manager  to  use  shortcuts  in  order  to  save  time.  Time  can  be  gained  in 
two  ways--first,  reduce  front-end  efforts  by  eliminating  front-end  training  analysis 
and,  second,  do  not  schedule  an  OTE  (Training  Evaluation). 

Indirect  requirements  are  equally  constrained  by  time.  When  the  requirement 
for  a device  is  established,  it  is  needed  then,  not  in  the  future.  This  causes 
the  acquisition  sponsor  to  take  every  possible  step  to  reduce  acquisition  time. 

The  elimination  of  OTE  is  one  obvious  method. 

POLICY  vs.  PRACTICE.  A discussion  of  existing  policy  vs.  practice  highlights 
deviations  in  the  training  device  acquisition  process  from  the  hardware  acquisition 
process.  Subsequent  paragraphs  discuss  the  major  considerations. 

Pol i cy.  D0D  policy  for  defense  systems  acquisitions  was  established  by  DOD 
Directive  5000.1.  This  policy  was  implemented  within  the  Navy  by  SECNAV  In- 
struction 5000.1.  The  actions  required  to  implement  policy  for  hardware  and 
operational  systans  are  detailed  in  0PNAVINST  3960.10.  No  reference  to  training 
devices  is  contained  in  either  policy  directive  or  the  implementing  instructions. 

It  is  logical  to  presume  that  training  devices  are  to  be  acquired  in  the  same 
manner  as  operational  hardware  and  systems. 

Mechanics  of  Implementation . 

1.  Operational  Systems.  The  mechanics  of  0PNAVINST  3960.10  are  strictly 
adhered  to  in  the  acquisition  of  operational  systems  and  equipments.  The  opera- 
tional test  agent  (OTA)  is  involved  with  the  program  from  the  Mission  Element 
Needs  Statement  (MENS)  or  Operational  Requirement  (OR).  A review  of  the  Navy 
Decision  Coordinating  Paper  (NDCP),  or  equivalent  document,  is  made  by  the  OTA 
and  comments  submitted  to  the  DA.  The  Test  and  Evaluation  Master  Plan  (TEMP)  is 
a joint  effort,  and  each  agent;  i.e.,  DA  and  OTA,  monitors  the  testing  of  the 
other  agent.  Reports  by  each  agent  are  scrutinized  by  the  other,  and  areas  of 
disagreement  are  settled  at  a conference  or  are  forwarded  to  higher  authority 

for  decision.  Approval  for  Service  Use  (ASU)  is  granted  using  both  the  Operational 
Evaluation  (0PEVAL)  and  Technical  Evaluation  (TECHEVAL)  as  major  inputs. 

2.  Training  Devices.  0PNAV1NST  3960.10  is  not  followed  in  the  acquisition 
of  training  devices  with  a minor  number  of  exceptions.  This  discussion  will  not 
address  the  exceptions,  rather  it  will  be  confined  to  the  general  procedures 
followed. 

a.  Devices  in  support  of  major  acquisition  programs,  direct  requirement, 
are  identified  by  the  DA  or  the  mission  sponsor.  No  attempt  is  made  to  define 
the  course  of  instruction,  to  identify  the  behavioral  objectives  of  the  course,  or 
to  establish  Job  Performance  Measures  (JPM)  prior  to  establishing  the  requirement 
for  a device.  A functional  specification  for  the  device  is  prepared  under  the 
auspices  of  the  DA.  The  Training  Comnand  is  brought  into  the  program  at  the 
Navy  Training  Plans  Conference  (NTPC),  by  which  time  the  training  device  is  well 
started  toward  acquisition.  No  attempt  is  made  to  develop  a TEMP  since  there 
is  no  designated  OTA.  A Fleet  Project  Team  is  formed  to  act  as  an  advisor  and 
to  assist  in  acceptance  testing.  This  team  is  responsive  to  the  DA,  the  fleet 
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commander,  and  the  mission  sponsor,  hence  is  not  independent,  nor  is  it  prepared 
to  act  as  experts  in  the  field  of  training.  Test  and  evaluation  is  confined  to 
technical  areas  and  is  restricted  to  an  assurance  that  the  device  meets  the 
technical  specifications.  There  is  only  an  acceptance  by  the  government  from 
the  contractor;  no  formal  ASU  procedures  are  implemented  before  follow-on  units 
are  procured. 

b.  Devices  required  in  support  of  other  than  major  acquisition 
programs;  i.e.,  indirect  requirements,  are  identified  by  the  training  agent. 

In  these  instances,  a course  of  instruction  and  learning  objectives  are  defined 
prior  to  the  decision  to  acquire  the  device.  A functional  specification  is 
prepared  by  the  training  agent  and,  from  this,  a technical  specification.  The 
acquisition  process  parallels  the  procedures  described  in  paragraph  a.  above 
through,  and  including,  acceptance.  Again,  there  is  no  independent  OTA;  conse- 
quently, no  OTE. 

Policy  Deviation  Causes.  Training  devices  differ  from  operational  equipments 
and  systems  in  a number  of  significant  ways  which  cause  them  to  be  viewed  as 
exceptions  to  the  general  policy.  The  major  differences  include  the  following: 

• Usually,  training  is  a subsystem  of  the  operational  system;  it  is 
supportive  of  the  operational  system  and,  therefore,  no  independent  T&E  is 
considered  needed. 

• Training  devices  to  be  maximally  effective  must  have  an  accompanying 
instructional  program.  Training  devices  support  a course  of  instruction,  and 
these  courses  support  the  operational  system. 

• Based  on  cost  alone,  most  training  devices  are  classified  ACAT  IV. 

Thus  in  terms  of  current  policy,  management  control  of  the  acquisition  process 
rests  with  the  Material  Command  rather  than  the  Training  Command. 

• Frequently,  the  Training  Command,  the  primary  user  of  training  devices, 
has  no  input  to  the  device  design  until  the  NTPC.  The  NTPC  for  major  hardware 
acquisitions  frequently  occurs  subsequent  to  contract  award.  In  total  package 
procurements  the  contract  includes  a training  package  (with  devices);  consequently, 
the  Training  Command  is  presented  with  a training  package  into  which  it  had 
little  or  no  official  input. 

• Conmander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR)  has 
stated  that  his  mission  does  not  include  nonoperational  hardware.  This  has  been 
supported  by  CNO  (OP-098).  There  is  no  designated  independent  operational  test 
and  evaluation  agent  for  shore-based  training  devices. 

MARINE  CORPS 

The  Marine  Corps  has  no  independent  T&E  policy  and  does  not  conduct  an 
independent  OTE.  Navy  policies  and  directives  are  applicable  since  the  Navy 
acquires  most  Marine  Corps  training  devices. 
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The  Army  organizational  structure  differs  from  the  Navy  structure  in  such  a 
way  that  the  conduct  of  T&E  cannot  be  paralleled.  The  Training  and  Doctrine 
Command  (TRADOC)  is  responsible  for  two  major  efforts--f i rst,  training  and, 
second,  the  establishment  of  operational  doctrine  which  inc  jdes  the  tactical 
use  of  hardware  and  personnel.  Because  of  this  dual  function  many  Army  requirements, 
both  operational  and  training,  originate  from  the  same  command.  TRADOC  Circular 
70-1  establishes  procedures  for  training  devices  T&E  of  nonmajor  acquisitions. 

Army  Regulation  71-3  (AR  71-3)  outlines  the  T&E  requirements  for  systems  designated 
as  major  by  the  Department  of  the  Army. 

Navy  training  is  concentrated  in  a few  commands,  and  each  of  these  commands 
instructs  in  a very  similar  manner  to  each  other.  Training  "is,  essentially, 
across  the  board.  Army  personnel  are  corps  trained,  and  training  is  assigned  to 
specific  forts,  each  fort  responsible  for  a given  type  of  training;  for  example. 

Fort  Rucker  does  all  aviation  training;  Fort  Knox,  armored  and  engineering;  Fort 
Bliss,  air  defense.  At  each  Army  training  establishment  there  is  a Board  or 
Test  Activity  which  is  responsible  to  TRADOC  for  the  test  and  evaluation  of  new 
training  devices.  However,  this  same  unit  is  also  involved  in  the  identifica- 
tion of  the  requirement.  Thus,  the  requirement  for  a device  is  defined  and 
its  subsequent  T&E  conducted  by  the  same  agent. 

Management  of  T&E  varies  with  the  designation  of  major  or  nonmajor  category. 
Those  designated  as  major  acquisitions  are  managed  by  the  Army  Operational  Test 
and  Evaluation  Agency  (OTEA);  nonmajor  acquisitions  are  managed  by  TRADOC  or  a 
TRADOC -designated  conmand.  In  both  instances  the  actual  test  plan  is  developed 
and  executed  by  the  Board  or  Test  Activity  at  the  fort.  Test  plans  are  approved 
by  the  test  manager,  and  these  same  managers  have  the  responsibility  for  monitor- 
ing the  execution. 

AIR  FORCE 

Air  Force  T&E  is  governed  by  Air  Force  Regulation  80-14  (AFR  80-14).  This 
document  and  OPNAVINST  3960.10  are  similar  in  their  requirements.  Training 
devices  in  general  are  not  discussed,  but  AFR  80-14  is  specific  in  the  require- 
ment to  conduct  an  independent  T&E  of  simulators,  even  one-of-a-kind. 

Discussions  with  Air  Force  personnel  revealed  that,  with  the  exception  of 
simulators,  there  is  no  policy  which  requires  an  independent  0TE  of  training 
devices,  and  none  is  contemplated  at  the  present  time. 
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SECTION  111 
TECHNICAL  APPROACH 

This  section  presents  the  approach  used  in  preparing  the  proposed  instruction 
together  with  a series  of  underlying  guidelines.  The  guidelines  were  established 
to  insure  that  the  T&E  of  training  devices  remained  compatible  with  existing 
procedures  for  the  acquisition  of  operational  hardware  and  systems.  How  these 
affect  the  draft  instruction  is  described  in  section  IV. 

To  accomplish  the  study  goals,  a rational  analytic  approach  was  employed. 

The  work  was  organized  in  two  phases: 

• Phase  1:  Investigated  the  current  factors  impinging  upon  training 
device  acquisition,  evaluation,  and  service  acceptance;  examined 
the  possibility  of  classifying  devices  for  evaluation  purposes; 
and  recommended  a concept  for  device  evaluation  and  acceptance 

(a  proposed  OPNAV  instruction). 

• Phase  2:  Based  on  the  phase  1 output,  prepare,  in  concert  with  OPNAV 
and  relevant  user  commands,  a comprehensive  OPNAV  instruction  to 
provide  for  the  effective  and  orderly  acceptance  of  all  Navy  training 
devices  acquired  with  RDT&E  funds. 

Implementation  of  this  work  effort  required  the  acquisition  of  information 
on  current  policies  and  practices  affecting  T&E  and  training  device  acquisition 
from  all  services.  The  following  specific  steps  were  taken: 

• review  of  policy  documents  from  DOD,  SECNAV,  CNO,  CNET,  and  the 
Chief  of  Naval  Material  (CNM)  to  provide  an  understanding  of  Navy 
policies  dealing  with  training  device  acquisition.  Navy  test  and 
evaluation,  and  service  acceptance 

• acquisition  of  information  front  those  various  Navy  commands/ 
activities  tasked  to  acquire,  evaluate,  and  accept  both  hardware 
and  training  devices  to  provide  an  understanding  of  how  policy  is 
translated  into  practice 

• acquisition  of  information  from  those  activities  involved  in 
the  budgeting  and  funding  of  training  devices  to  insure  that 
financial/fiscal  policies  are  compatible  with  T&E  requirements 

• review  of  policy  documents  and  the  acquisition  of  information 
from  other  services  (Marine  Corps,  Army,  Air  Force)  charged 
with  the  acquisition  and  T&E  of  training  devices  to 
identify  potential  applications  for  Navy  use 

• acquisition  of  information  from  the  COMOPTEVFOR  to  acquire  an  under- 
standing of  the  specific  techniques  used  in  the  Navy  T&E  of  hardware 
and  to  identify  potential  application  of  those  techniques  to  training 
devices. 
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The  information  and  data  gathered  during  these  steps  were  used  to  establish 
training  device  T&E  requirements  and  to  identify  those  factors  which  affect 
training  device  T&E  implementation.  A technique  for  classifying  training 
devices  for  test  and  evaluation  and  a methodology  for  the  future  T&E  and 
service  acceptance  of  training  devices  were  developed.  Also,  a process  flow 
which  integrates  all  of  the  elements  affecting  training  device  acquisition  and 
T&E  was  developed  to  facilitate  functional  compatibility  with  current  acquisition 
and  fiscal  procedures. 

STUDY  GUIDELINES 

During  the  development  of  the  technical  approach,  it  became  obvious  that 
limitations  to  the  development  effort  were  necessary  to  preclude  unwarranted 
deviation  from  existing  practice.  As  a control  the  following  three  guidelines 
were  established. 

RDT&E  FUNDING  GUIDELINES.  With  the  Comptroller  of  the  Navy  (NAVCOMP)  decision 
that  the  first  article  or  prototype  training  devices  be  procured  with  RDT&E 
funds,  two  major  problems  became  apparent  and  required  resolution  in  the  proposed 
instruction.  First,  since  training  devices  generally  fall  in  the  ACAT  IV  category 
money  constraints,  the  management  responsibility  for  acquiring  these  devices 
rests  with  a Navy  Systems  Command  (SYSCOM).  SYSCOMs  are  oriented  toward  engineering, 
design,  managerial,  and  contractual  functions  and  have  little  expertise  in  the 
training  and  education  field.  Second,  the  cost  of  the  time  to  accomplish  T&E, 
particularly  an  independent  OTE,  could  cause  excessive  delays  in  the  training 
device  acquisition  process.  To  accommodate  these  two  problems,  the  following 
rules  were  established: 

• The  training  and  educational  development  process  which  leads  to  new 
training  devices  must  insure  that  device  requirements  are  no  more  nor 
less  than  is  actually  required.  Both  over  and  underdesign  of  training 
devices  are  expensive  and  inefficient. 

• Training  device  OTE  should  be  planned  for  and  conducted  by  persons  who 
possess  practical  expertise  in  the  education  field.  This  will  give  a 
high  probability  that  the  test  and  evaluation  will  be  conducted  quick- 
ly using  minimum  resources. 

• The  application  of  established  practices  used  in  hardware  T&E  to 
training  device  T&E  will  establish  a better  working  relationship  and 
understanding  between  the  hardware  acquisition  command  and  the  training 
community. 

• The  development  of  detailed  implementation  procedures  for  the  T&E 
of  training  devices  must  be  the  responsibility  of  the  activity 
designated  to  perform  the  actual  evaluation.  The  proposed  instruction 
will  be  restricted  to  the  general  procedures  necessary  to  satisfy 

the  CNO  requirements  and  established  DOD  policy. 

• The  draft  instruction  must  recognize  the  basic  element  underlying 
the  design,  development,  acquisition,  and  use  of  any  training  device; 
a training  device  does  not  exist  in  and  of  itself  but  rather  that 
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it  exists  to  support  a training  requirement.  Training  devices  cannot 
be  designed  or  evaluated  in  a vacuum,  rather  in  every  instance,  they 
perform  a supporting  role  in  a total  training  system.  For  T&E,  this 
means  that  assessment  of  effectiveness  is  accomplished  not  only  in 
terms  of  what  the  device  can  do  but  also  in  terms  of  how  well  it 
supports  identifiable,  required  training.  Thus,  a complete  T&E  of  a 
training  device  requires  the  existence  of  stated  training  requirements, 
usually  provided  in  the  form  of  course  behavioral  objectives  and  JPM. 

EXISTING  PROCEDURES  GUIDELINES.  The  Navy  follows  a well  established,  proven 
series  of  procedures  in  the  T&E  of  operational  hardware  and  systems.  These  are 
in  consonance  with  established  policy  and  are  thoroughly  understood  by  both 
acquisition  managers  and  test  personnel.  In  the  interest  of  efficiency,  it  is 
considered  highly  desirable  to  emulate  these  procedures  whenever  possible.  To 
insure  this: 

• there  must  be  a minimum  deviation  from  existing  policy  for  operational 
hardware  and  systems 

• there  must  be  a high  degree  of  management  flexibility  incorporated 
into  the  T&E  process 

• the  implementation  of  the  T&E  process  must  maintain  the  established 
chain  of  command 

• the  T&E  policy  must  be  compatible  with  the  existing  POM/budget  cycles 

• the  use  of  resources;  i.e.,  time,  personnel,  and  money  must  be  held  to 
a minimum 

• established  procedures  for  the  Development  Test  and  Evaluation  (DTE) 
of  training  devices  require  no  modification  in  substance  from  the 
procedures  established  for  operational  systems  or  equipments.  These 
procedures  have  served  well  and  are  regularly  updated  as  a matter  of 
course.  Specific  tests  that  make  up  the  DTE  portion  of  T&E  remain  the 
purview  and  responsibility  of  the  DA. 

PACING  FUNCTION  GUIDELINES.  Current  policy  directives  prescribe  that  the  achieve- 
ment of  acquisition  program  milestones,  which  include  T&E,  is  the  pacing  function 
that  drives  both  the  program  structure  and  resource  allocation.  This  series  of 
published  milestones  is  equally  applicable  to  operational  equipment/systems  and 
training  devices.  This  means  that  resources  will  be  allocated  and  the  acquisitions 
so  timed  that  no  critical  steps  will  be  omitted. 
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SECTION  IV 

FACTORS  INFLUENCING  THE  OPNAV  INSTRUCTION 

This  section  presents  in  detail  the  major  factors  which  must  be  accounted 
for  in  the  proposed  OPNAV  instruction.  The  factors  are  organizea  into  four 
primary  areas,  each  of  which  is  discussed  in  turn.  The  four  areas  are  policy, 
management,  the  POM/budget  cycle,  and  device  integration  into  the  overall 
acquisition  system. 

POLICY 

Policy  refers  to  the  underlying  structure  in  RDT&E  acquisition  programs. 

This  emphasizes  a minimum  deviation  from  existing  operational  hardware  acquisition 
procedures. 

CHAIN  OF  COMMAND.  The  Navy  establishes  and  assigns  responsibility,  authority, 
and  accountability  through  an  organizational  hierarchy.  Implementation  of  a 
process  for  test  and  evaluation  of  Navy  training  devices  should  maintain  the 
specific  elements  of  command  chains.  If  this  is  not  done,  then  the  relative 
placement  of  the  elements  must  be  sequenced  so  that  they  provide  for  the  realistic 
management  and  integration  of  that  process  into  the  Navy  as  a whole. 

Implications  of  this  for  the  recommended  T&E  process  include  the  following: 

• specific  definition  of  the  responsibility/function  of  all  activities 
involved  in  training  device  T&E 

• establishing  a organizational  structure  which  will  not  violate 
currently  operating  echelons  and  normal  chains  of  command 

• placing  the  OTA  in  the  training  organization  as  an  independent  agency 
but  responsive  through  the  established  hierarchy. 

DOCUMENTATION.  The  need  for  adequate,  timely,  and  accurate  documentation  of 
events  and  actions  in  the  acquisition  and  T&E  process  cannot  be  overemphasized. 
Required  documents  are  identified  in  the  proposed  instruction,  and  various 
enclosures  and  tabs  to  the  instruction  provide  sample  formats.  Required 
documents  include: 

• MENS,  or  equivalent 

• NDCP,  or  equivalent 

• Requirement(s)  Statement(s) 

• JPMs/Behavioral  Objectives 

• Device  Test  and  Evaluation  Master  Plan  (DTEMP) 

• Device  Test  Plan 

• Test  Results  Reports. 
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T&E  SELECTION.  Training  devices  exist  in  a variety  of  sizes,  capabilities,  and 
costs.  Accordingly,  different  levels  and  types  of  T&E  are  required.  The 
application  of  existing  ACAT  categories  to  training  device  T&E  is  not  fully 
satisfactory.  For  example,  all  ACAT  III  acquisitions  are  required  to  have 
independent  evaluations,  and  ACAT  IV  acquisitons  are  managed  outside  of  the 
Training  Command.  A method  of  categorizing  training  devices  was  developed  to 
select  those  in  the  ACAT  III  and  ACAT  IV  categories  for  which  the  operational 
portion  of  T&E  must  be  accomplished  by  an  independent  agent,  and  those  for  which 
operational  testing  can  be  accomplished  by  the  DA. 

Cost  alone  cannot  be  the  sole  criteria  for  T&E  selection.  Some  relatively 
inexpensive  devices  are  designated  ACAT  III  in  order  to  maintain  acquisition 
managerial  control  within  the  Training  Command.  This  situation  is  particularly 
applicable  for  devices  acquired  because  of  indirect  requirements.  Yet,  these 
same  training  devices  may  not  warrant  the  time  and  resource  expenditure  necessitated 
by  an  independent  OTE.  A system  is  needed  whereby  managerial  control  remains 
within  the  Training  Command,  yet  an  independent  OTE  is  not  required.  Conversely, 
some  ACAT  IV  training  devices  may  be  of  such  importance  that  they  require  an 
independent  OTE. 

RDT&E  support,  whether  Fleet  or  Other,  is  sometimes  required  in  the  execution 
of  T&E.  This  support  is  best  obtained  using  the  DTEMP  rather  than  through  other 
means.  Inexpensive  devices  in  the  ACAT  IV  category  which  require  RDT&E  support 
should  also  involve  this  simple  method. 

To  accommodate  situations  arising  from  T&E  selections.  Device  Categories 
(DCAT)  which  apply  only  to  ACAT  III  and  IV  training  device  acquisitions  have 
been  developed  by  TAEG.  These  DCATs  permit  the  Assessment  Sponsor  to  determine 
whether  a training  device  requires  an  independent  OTE  or  the  submission  of  a 
DTEMP.  The  categories,  as  included  in  the  proposed  OPNAV  instruction,  are  shown 
in  table  1. 


TABLE  1.  DEVICE  CATEGORIES  (DCAT) 


DCAT 

A* 


Designated  By 
Assessment  Sponsor 


Nominal  Dollar  Value  Thresholds 
RDT&E  Production 


$1.5M  $ 5M  Other  programs  recommended 

by  CNET,  OP-098,  or  DA 


B Assessment  Sponsor 


Training  devices  which 
interface  directly  with 
operational  systems  and 
require  OTE  with  that 
system 


* 


C*  CNET  All  programs  not 

designated  DCAT  A or  B 


Programs  below  the  DCAT-A  dollar  threshold  will  normally  be  designated  DCAT-A  if: 


a.  They  require  an  independent  OTE  to  support  key  program  decisions,  or 

b.  They  require  Fleet  or  Other  RDT&E  Support. 
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MANAGEMENT 

A high  degree  of  flexibility  is  regained  in  the  T&E  process  to  insure  that 
training  device  acquisitions  are  timely  and  that  minimum  resources  are  expended. 
The  management  functions  which  directly  affect  this  flexibility  are  discussed 
below. 

RESOURCES.  Policy  established  for  the  T&E  of  training  devices  must  reflect 
practical  considerations.  Two  specific  resource  constraints  that  have  the 
greatest  affect  on  the  T&E  process  are  time  and  the  use  of  fleet  and  other 
support  personnel/equipment  in  the  process. 

In  general,  the  time  available  for  training  device  acquisition  T&E  is 
compressed  in  comparison  to  counterpart  hardware  buys.  For  example,  the  develop- 
ment of  a new  simulator  to  support  training  for  a new  weapons  system  presumes 
the  identification  of  related  course  requirements  and  behavioral  objectives 
prior  to  the  decision  to  acquire  the  device.  In  current  hardware  acquisition 
practice,  behavioral  learning  objectives  are  not  established  until  about  Milestone 
II  (the  Engineering  Development  decision  point)  at  the  NTPC.  Usually,  this 
precedes  system  acceptance  by  3 years.  Only  3 years  are  available  in  which  to 
design  and  develop  the  device  and  to  conduct  the  T&E.  This  is  considerably  less 
time  than  is  available  for  conceptualizing,  designing,  and  conducting  T&E  of 
operational  hardware.  Thus,  managers  tend  to  use  device  acquisition  methods 
which  save  time  irrespective  of  optimum  efficiency. 

The  use  of  fleet  and  other  support  personnel  and  equipment  for  the  T&E  of 
hardware  can  be  justified  by  the  mission,  by  the  cost,  or  by  numbers  of  the 
equipment  itself.  Similar  kinds  of  manpower  and  equipment  for  training  device 
T&E  is  not  so  easily  supported  in  the  competition  for  these  resources.  The 
reason  is  that  most  often  devices  are  acquired  in  small  numbers,  cost  less,  and 
provide  only  one  means  of  instructional  delivery. 

SCOPE  OF  OPERATIONAL  TEST  AND  EVALUATION.  The  application  of  OTE  to  training 
devices  is  simply  an  extension  of  existing  technical  testing.  The  extension  is 
the  application  of  operational  considerations  to  the  device  in  addition  to 
presently  required  technical  test  and  evaluation.  Operational  testing  of 
training  devices,  however,  is  more  limited  in  scope  than  that  required  for  the 
operational  testing  of  hardware.  What  is  intended  under  OTE  procedures 
is  the  determination  of  the  training  effectiveness  and  suitability  of  training 
device  performance  as  it  supports  a training  system  in  its  intended  training 
environment.  OTE  of  a training  device  is  not  intended  to,  nor  can  it,  measure 
how  well  the  training  system  supports  the  transfer  of  training  that  may  take 
place  from  the  training  system  to  the  operational  system.  It  is  probable  that 
findings  related  to  areas  other  than  device  effectiveness  (i.e.,  new  training 
strategies)  will  result  from  OTE.  Obviously,  these  findings  are  of  importance 
to  the  training  community  and  should  be  communicated.  However,  for  purposes  of 
device  OTE,  the  evaluation  is  restricted  to  how  well  the  device  supports  Its 
program  of  training. 

ROT&E  SUPPORT.  Training  device  development,  generally,  does  not  require  Fleet 
RDT&E  Support.  Rather,  this  development  effort  requires  support  from  activities 
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ashoru;  r.g.  , schools,  ranges,  and  laboratories  . I herd  ore,  there  should  be  the 
capability  of  requesting  this  support  in  a manner  similar  to  requesting  fleet 
support.  The  ashore  support  should  be  included  in  the  DTEMP.  In  this  manner, 
all  parties  involved  are  made  aware  of  the  requirement  and  can  plan  accordingly. 

As  with  existing  practice,  this  support  would  necessitate  CNO  approval . In  the 
proposed  instruction,  this  requirement  has  led  to  the  inclusion  of  two  classes 
of  RDT&E  Support:  fleet  and  Other,  fleet  RDT&E  Support  is  provided  by  opera- 
tional Naval  forces,  and  Other  RDT&E  Support  is  provided  by  any  command,  activity, 
or  agent  not  under  the  direct  operational  command  of  a fleet  commander. 

COMOPTEVfOR  INVOLVEMENT.  COMOPTEVfOR  has  stated  that  his  mission  is  concerned 
with  operational  hardware  and  systems  supplied  to  the  operational  forces.  His 
responsibilities  do  not  include  training  devices  developed  for  use  at  shore 
installations.  However,  there  are  certain  training  devices  which  have  been 
developed  to  interface  directly  with  operational  equipment,  and  stimulate  this 
equipment  for  training  purposes.  Thus,  any  OTE  of  this  class  of  device  would 
have  to  be  conducted  in  conjunction  with  operational  equipment  and  could  affect 
the  operational  capabilities  of  the  system/equipment.  To  insure  the  device  does 
have  satisfactory  operational  suitability,  it  is  proposed  that  these  types  of 
devices  be  tested  and  evaluated  by  COMOPTEVfOR  with  the  educational  and  training 
expertise  provided  by  CNET  when  requested.  This  would  have  the  benefit  of 
making  requisite  educational  expertise  available  to  COMOPTEVfOR  without  the 
necessity  of  augmenting  the  staff. 

COORDINATION  Of  TRAINING  DEVICE  AND  POM/BUOGET  PROCESSES 

Two  separate  POM/budget  cycles  must  be  considered  in  the  acquisition 
process  of  training  devices.  R&D  funding  is  used  for  initial  unit  or  prototype 
development  and  includes  provisions  to  support  T&E  requirements.  These  require- 
ments specify  the  manpower,  money,  and  facilities  support  needed  to  accomplish 
the  test  plan,  follow-on  units  are  acquired  using  different  funding,  and  this 
implies  a need  for  sufficient  early  planning  within  individual  resource  sponsor 
POM/budgets  to  enable  the  timely  acquisition  of  money.  The  coordination  of  fund 
expenditures  from  different  sources  necessitates  maximum  communi cation  and 
interaction  among  all  activities  involved,  and  the  T&E  process  that  is  established 
must  provide  for  this  need. 

R&D  fUNDING.  R&D  funding,  which  is  mandated  for  specified  devices  meeting 
applicable  criteria,  covers  a period  in  time  from  device  initiation  to  "formal 
Navy  acceptance  of  the  device."  No  R&D  monies  will  be  allocated  for  device 
procurement  following  such  acceptance,  which  is  defined  as  coincident  with  ASU. 
This  point  is  discussed  subsequently  under  Integration  of  the  Training  Device 
Acquisition  Process  into  the  Overall  System. 

Existing  procedures  for  the  allocation  of  R&D  monies  parallel  closely  the 
procedures  used  to  fund  development  of  hardware  systems  and  equipments.  Because 
of  this,  specific  explanation  of  the  R&D  budget  cycle  is  not  required  here.  The 
establishment  of  milestones  for  the  reviews  of  program  objective  accomplishment 
occurs  as  a normal  part  of  the  R&D  funding  process.  T&E  budget  requirements 
reflect  T&E  needs  spelled  out  in  the  DTEMP  and  supporting  documents  and  are 
closely  tied  to  the  established  milestones  and  reviews.  All  usual  stipulations 
for  use  of  R&D  monies  will  apply  during  prototype  acquisition. 
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It  is  proposed  that  R&D  budget  submissions  be  made  coincident  with  hard- 
ware/system  milestone  1 for  direct  training  device  requirements.  These  submis- 
sions can  be  updated  on  an  annual,  or  on  an  as  occurring  basis.  Authority  to 
expend  would  be  granted  when  the  decision  is  made  that  a new  training  device  is 
required. 

Indirect  training  device  requirements  present  a problem  with  respect  to 
funding.  Recognition  of  the  need  for  a device  occurs  when  the  requirement  is 
identified.  It  is  at  this  time  that  the  budget  submission  would  be  made. 

Expenditure  authority  would  occur  at  the  same  relative  point  in  time  as  for 
direct  requirements. 

PRODUCTION  UNIT  FUNDING.  The  funding  of  production  units  of  a training  device 
is  the  responsibility  of  sponsors.  While  these  sponsors  may  vary,  depending 
on  the  type  of  training  device  and  its  intended  use,  it  is  essential  that  all 
training  device  follow-on  units  be  planned  and  budgeted  in  a careful  and 
timely  fashion  to  preclude  significant  acquisition  delays. 

Figure  1 shows  a general  flow  of  budget  events  leading  to  funding  execution 
for  a fiscal  year.  Working  from  the  beginning  of  the  fiscal  year  being  executed 
back  to  the  initial  NAVCOMP  Budget  Call,  it  can  be  seen  that  Congress,  DOD, 
and  NAVCOMPT  reviews  require  that  budget  preparation  begin  at  least  18  months 
before  execution.  Under  normal  circumstances,  this  means  that,  for  follow-on 
units  of  a training  device  to  be  procured  during  a given  fiscal  year,  planning 
for  budget  insertion  must  occur  prior  to  the  first  review,  a period  of  from  l1* 
to  2S  years  prior  to  the  time  funds  are  expected.  To  wait  until  ASU  is  complete 
before  budget  insertion  would  create  a l'a  to  2l2  year  delay  between  prototype  and 
follow-on  units,  an  unacceptable  situation. 

Resolution  of  the  budget-acquisition  dichotomy  lies  in  the  establishment 
of  a policy  which  continues  to  recognize  the  ASU  requirement  before  an  expenditure 
authorization  but  allows  a budget  insertion  based  on  early  T&E  results.  A compari- 
son of  course  objectives  with  the  training  device  design  is  the  first  operational 
test  associated  with  the  course  and  device  development  efforts.  For  course  develop- 
ment, JPMs/behavioral  objectives  have  been  established;  for  the  device,  a design 
freeze,  or  its  equivalent,  has  occurred,  and  device  design  characteristics  have 
been  formally  agreed  upon.  The  application  of  OTE  at  this  point  takes  the  form 
of  a review  of  projected  device  characteristics  and  technical  specifications 
measured  against  behavioral  objectives  established  by  course  developers.  Satisfactory 
completion  of  this  evaluation,  identified  as  OT -II,  serves  as  justification  to 
insert  budget  entries  requesting  allocation  of  funds  for  follow-on  units.  It  is 
emphasized  that  only  budget  insertion  is  justified  at  this  point;  expenditure  of 
funds  requires  ASU,  or  a waiver,  and  authority  to  expend,  which  is  based,  in 
part,  on  completion  of  DT /OT -III. 

The  implementation  of  this  policy  will  result  in  a defined  relation  between 
the  budget  and  acquisition  cycles.  For  DCAT  A devices,  a minimum  of  18  months 
is  required  from  DT/OT-II  to  availability  of  monies  for  follow-on  units.  During 
this  period,  prototype  device  procurement  is  completed,  final  course  development 
is  accomplished,  and  T&E,  through  TECHEVAL/TRAEVAL  (DT/OT-II I ) , Is  conducted. 

The  establishment  of  18  months  as  the  norm  does  not  mean  that  other  budget 
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"windows"  lire  not  avai  l.tblo  tot-  tol  low-on  unit  insertion.  In  instances  where 
the  priority  of  the  need  for  a particular  training  device  is  high  enough  to 
warrant  deviation  from  the  norm,  current  budget  insertion  “windows"  occur  later 
in  the  cycle  and  can  be  used  to  insert  money  into  the  budget  for  additional 
units.  To  illustrate,  the  CNET  budget  cycle  normally  includes  an  apportionment 
conference  about  6 months  prior  to  budget  execution,  during  which,  with  justifi- 
cation, funds  assigned  to  one  project  or  program  could  be  reallocated  to  accommo- 
date some  more  pressing  priority.  In  those  instances  of  high  priority,  or  time 
constraint,  a procedure  exists  by  which  T&E  can  be  waived  as  a requirement  for 
the  production  decision.  Such  a waiver  does  not  obviate  the  requirement  for 
T&E;  rather,  it  postpones  it  until  some  point  after  production  has  started. 

One  basic  difficulty  related  to  budget-acquisition  integration  remains  and 
is  presented  as  an  issue  for  consideration  in  section  V of  this  report.  This 
difficulty  involves  the  satisfaction  of  lead  time  requirements  for  MILCON  funding 
where  congressional  approval  is  necessary.  In  most  cases,  5 years  has  been 
stipulated  as  a minimum  lead  time  for  MILCON  budget  insertion.  No  solution  to 
this  difficulty  has  been  found. 

INTEGRATION  OF  THE  TRAINING  DEVICE  ACQUISITION  PROCESS  INTO  THE  OVERALL  SYSTEM 

Since  training  devices  are  conceived,  designed,  developed,  and  procured  to 
support  a training  course,  it  is  logical  to  presume  the  existence  of  the  course 
prior  to  device  development.  This  is  not  always  the  case.  However,  for  the  OTE 
of  training  devices  the  existence  of  the  course,  or  at  least  its  behavioral 
objectives  and  evaluation  criteria,  is  mandatory  prior  to  the  decision  to  procure. 

The  problem  associated  with  the  timing  of  the  device  funding  and  the  POM/ 
budget  cycle  must  be  accommodated.  Coordination  must  be  accomplished  in  such  a 
way  that  inordinate  delays  are  not  encountered.  Figure  2 is  the  recommended 
training  system  and  training  device  process  flow  as  it  relates  to  hardware 
milestones  and  the  budget  process.  The  milestone  events  and  the  various  stages 
of  T&E  are  depicted  at  the  approximate  relative  time  they  must  occur  if  there  is 
to  be  no  delay. 

The  following  paragraphs  depict  the  events  in  figure  2 as  they  relate  to  a 
training  device  acquired  to  satisfy  a direct  requirement.  Indirect  requirements 
are  established  through  a similar  process,  except  that  there  is  no  consideration 
of  hardware  milestones.  Rather,  the  training  device  milestones  would  be  the 
critical,  driving  force. 

Event  1:  Identify  Requirements.  The  requirements  for  a new  training  device  can 
come  from  a variety  of  sources.  For  example,  within  the  NAVEDTRACOM,  school  or 
support  personnel  may  identify  such  requirements.  Outside  of  the  training 
establishment,  fleet  personnel  may  initiate  ideas.  Whatever  the  source, 
training  device  requirements  can  be  grouped  into  one  of  two  categories: 

• those  required  to  support  new  or  modified/revised  learning 
requirements,  necessitating  the  developing  of  new  course/ 
behavioral  objectives 

• those  which  support  a continuing  training  system. 
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Figure  2.  Recommended  Training  Device  Acquisition  Proi 
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Within  the  acquisition  flow,  the  identification  of  requi rements  provides 
the  first  step  in  the  T&E  process  in  that  requirements  provide  the  criteria 
against  which  the  effectiveness  of  the  device  will  eventually  be  measured. 

The  identification  of  the  training  requirement  is  a natural  result  of 
hardware  milestone  1 and  should  compel  the  Resource  Sponsor  to  make  a preliminary 
identification  of  RDT&E  funds  which  may  be  needed  in  the  acquisition  of  a prototype 
training  device.  This  is  milestone  0 for  the  training  system  and  device. 

Event  2:  Requirements  Validation.  In  addition  to  actually  verifying  a training 
support  need  that  may,  or  may  not,  include  a training  device,  actions  accomplished 
during  this  event  serve  to  identify  those  commands  and  activities  w :h  will 
be  involved  in  the  acquisition. 

Event  3:  New  Courses/Behavioral  Objectives  Decision  Point.  This  event  determines 
the  level  of  course  development  effort  required  to  establish  evaluation  criteria. 

If  a new  course  is  necessary,  the  design  and  development  effort  continues  through 
events  3A,  3B,  and  3C  of  figure  2.  If  not,  the  process  proceeds  to  event  4, 
with  agreement  among  all  concerned  that  existing  behavioral  objectives  will 
suffice  to  define  the  training  requirements  against  which  a new  device  may  be 
acquired  and  evaluated. 

The  th -ee  events  related  to  course  development  are  not  all  the  actions  required 
to  fully  implement  a new  program  of  study.  However,  for  a discussion  of  T&E, 
these  three  events  provide  the  essential  framework  for  the  test  and  evaluation 
of  training  devices  developed  to  support  the  course. 

Assuming  the  need  for  a new  course  or  the  formulation  of  new  behavioral 
objectives,  course  design  (event  3A)  is  taken  to  the  point  that  an  initial 
review  of  the  objectives  can  be  conducted  to  verify  that  they  satisfy  validated 
fleet  training  requirements  (event  2).  Whether  an  Instructional  Systems  Develop- 
ment (ISD)  or  another  process  is  used  as  a guide,  it  is  at  this  point  that  initial 
behavioral  objectives  with  JPMs  must  be  developed  for  comparison  with  the  validated 
requirement  (event  3B).  These  objectives  with  JPMs  serve  as  the  criteria  against 
which  a preliminary  design  of  the  training  device  can  be  reviewed  (event  4B)  at 
some  later  time.  Timing  of  the  development  of  initial  objectives  is  important 
as  thq  review  (event  4B)  must  occur  at  or  before  the  DT/OT-II  point  in  the  T&E 
cycle. 

A satisfactory  result  from  this  preliminary  course  evaluation  (event  3B) 
leads  to  continuation  and  completion  of  the  course  development  effort  (event 
3C).  Course  development  is  not  tied  to  the  training  device  acquisition  or  T&E 
cycle  other  than  that  course  completion  must  occur  before  OT -III  of  the  device. 

Event  4:  Training  Device  Decision  Point.  This  event  initiates  the  actual 
acquisition  cycle  for  the  device.  In  addition  to  this  critical  feature,  its 
position  in  the  acquisition  cycle  is  critical;  i.e.,  after  the  course  decision 
has  been  made.  This  positioning  serves  to  highlight  the  supportive  nature  of  a 
device  in  the  training  system.  The  decision  to  acquire  a new  training  device 
leads  to  the  acquisition  events  (events  4A  through  4D)  involved  in  that  process. 
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lor  l&L,  another  crucial  result  of  art  affirmative  decision  to  acquire  a 
device  (event  4)  is  the  requirement  to  initiate  appropriate  documentation.  Of 
greatest  importance  is  the  DTEMP,  the  primary  coordination  document  for  the  T&E 
process.  The  DTEMP  includes  a preliminary  statement  of  what  T&E  is  required, 
estimates  the  resources  required  to  support  the  T&E,  establishes  time  schedules, 
and  confirms  the  specific  assignments  of  personnel  and  activities  involved,  to 
include  RDT&E  Support.  Initial  evaluation  criteria  for  both  development  and 
operational  tests  are  specified.  The  DTEMP  is  initiated  as  early  in  the  train- 
ing device  acquisition  process  as  possible. 

Within  the  training  device  acquisition  process  (events  4A-4D),  the  two 
decision  points  (events  4B  and  4D)  are  most  important  from  the  perspective  of 
T&E.  The  first  of  these  (event  4B)  represents  that  point  in  the  acquisition 
cycle  when  the  device  reaches  contract  design  freeze  or  its  equivalent.  Design 
work  on  the  proposed  device  has  reached  the  stage  where  further  changes  will  be 
minimal,  and  proposed  device  operating  characteristics  and  specifications  can  be 
evaluated  against  the  initial  behavioral  objectives  and  JPMs.  This  evaluation 
(DT/OT-II)  is  a paper  analysis  and  results  in  a report  which,  assuming  that  the 
results  are  satisfactory,  provides  the  justification  for  insertion  of  follow-on 
unit  requirements  into  the  appropriate  budget. 

The  second  decision  point  identified  in  this  flow  (event  4D)  is  the  final 
major  T&E  event.  At  this  point,  the  prototype  device  is  tested  against  the 
criteria  provided  by  the  fully  developed  course  objectives  and  the  results 
evaluated  in  terms  of  the  validated  requirements  (event  2).  Event  4D  is  DT/OT- 
III,  TECHEVAL/TRAEVAL , the  major  T&E  event  that  occurs  during  any  device  acquisi- 
tion process,  and  it  provides  a major  input  to  the  ASU  decision  (event  5). 

The  training  system  is  now  available  to  the  hardware  developers  for  use  in 
initial  training  of  the  individuals  assigned  to  conduct  the  OPEVAL/TECHEVAL  of 
the  hardware.  Thus  the  effectiveness  of  the  system  can  be  verified  in  the 
actual  operational  hardware  environment. 

Event  5:  Approval  for  Service  Use  Decision  Point.  The  ASU  event  is  the  final 
verification  that  the  device  meets  operational  needs  prior  to  expenditure  of 
funds  for  follow-on  units.  As  in  hardware  acquisition,  it  is  a prerequisite  to 
the  production  decision.  ASU  depends  on  a variety  of  inputs,  including  the 
results  of  DT/OT -III.  Although  funds  for  follow-on  units  were  inserted  in  the 
budget  at  DT/OT-II,  no  authority  to  expend  had  been  given.  ASU,  or  a waiver  as 
provided  in  the  proposed  OPNAV  instruction  enables  the  Acquisition  or  Resource 
Sponsor  to  authorize  the  expenditure  of  budgeted  funds  at  the  third  device 
milestone  (Ml-3). 

The  major  T&E  events  of  training  device  acquisition  have  been  designed  to 
flow  smoothly  from  one  event  to  the  next  providing  ample  opportunity  for 
coordination  among  responsible  commands.  The  events  are  mutually  supportive. 
Although  each  event  in  figure  2 is  identified  separately,  it  is  probable  that 
many  events  will  occur  simultaneously.  For  example,  the  Validation  of  Require- 
ments (event  2),  the  New  Course  Decision  Point  (event  3),  and  the  Training 
Device  Decision  Point  (event  4),  could  occur  as  a result  of  decisions  reached  at 
a single  convened  conference.  Additionally,  the  development  of  a new/revised 
course  and  the  acquisition  of  the  prototype  device  could  occur  during  the  same 
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period  of  time.  Specific  coordination  efforts  would  be  required  among  course 
and  device  development  personnel  to  ensure  that  the  elements  needed  for  test  and 
evaluation  are  accomplished  in  a timely  fashion. 
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SECTION  V 

POLICY  ISSUES  FOR  CNET  CONSIDERATION 

During  the  development  of  the  proposed  instruction  on  Test  and  Evaluation 
of  Training  Devices,  major  questions  of  policy  and  procedure  were  raised,  the 
resolution  of  which  required  the  experienced  judgment  of  professionals  represent- 
ing involved  commands.  Where  appropriate,  opinions  and  recommendations  were 
solicited  from  persons  at  these  activities.  These  responses  were  considered  in 
arriving  at  the  specific  recommendations  made  in  the  proposed  instruction. 
Nevertheless,  the  policies  and  procedures  recommended  require  the  concurrence 
and/or  approval  of  conmands  responsible  for  them. 

This  section  presents  those  issues  raised  for  consideration  and  approval. 
Amplifying  remarks  are  included  to  highlight  factors  affecting  any  specific 
issue. 

Issue  No.  1 . Training  devices  will  be  acquired  using  as  nearly  as  possible  the 
same  criteria  as  operational  systems  and  operational  hardware. 

Acceptance  of  this  premise  will  require  an  extensive  course  development 
effort  prior  to  the  decision  to  acquire  a training  device.  As  a corollary  to 
this  acceptance,  the  formal  inclusion  of  training  command  representatives  will 
be  required  much  earlier  in  operational  systems  acquisitions  than  is  presently 
the  case. 

Issue  No.  2.  The  DCAT  categorization  is  necessary  to  insure,  within  existing 
policy,  that  the  acquisition  of  training  devices  is  managed  by  a command  with 
the  requisite  expertise,  and  that  the  conduct  of  an  independent  OTE  is  restricted 
only  to  devices  of  major  impact. 

It  is  not  necessarily  efficient,  or  cost  effective,  to  faithfully  simulate 
operational  equipment  for  training.  Since  good  training  decisions  on  fidelity 
of  simulation  and  the  extent  of  simulation  require  considerable  skills  and 
experience,  the  management  control  of  the  device  acquisition,  particularly 
design,  should  reside  with  the  training  experts.  The  criteria  for  using  RDT&E 
funds  for  prototype  development  are  identical  for  expensive  devices  and  for 
inexpensive  devices.  It  is  not  necessary  to  conduct  independent  OTEs  of  compara- 
tively minor,  inexpensive  training  devices,  particularly  since  the  cost  of  these 
evaluations  in  terms  of  time  and  resources  could  exceed  device  cost. 

Issue  No.  3.  Assign  CNET  the  responsibility  for  the  management  of  the  conduct 
of  OTE  for  training  devices  and  for  the  appointment  of  an  OTA  within  the  Training 
Command. 

This  concept  results  from  the  nature,  use,  and  location  of  training  devices. 
There  is  no  existing  organization  staffed  with  adequate  numbers  of  educational 
specialists  whose  mission  includes  the  conduct  of  the  OTE  of  training  devices. 

The  logical  location  of  such  an  agent  is  within  the  command  where  the  majority 
of  training  experts  are  located.  Acceptance  of  this  concept  has  two  major 
advantages. 
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1.  II  rtrUiMr.  the  OIA*.  independence  and  the  OPNAV  approval  authority 
for  the  conduct  of  OTE. 

2.  It  precludes  the  need  for  augmenting  the  COMOPTEVTOR  staff  and  expand- 
ing his  mission. 

Issue  No.  4 . Differentiate  between  Fleet  and  Other  RDT&E  Support  for  the  develop- 
ment of  training  programs  and  the  conduct  of  training  device  OTE. 

Training  devices  are  designed  in  a nonoperational  environment.  Research 
support  for  programs  supported  by  training  devices  is  generally  required  from  the 
shore  establishment  rather  than  the  fleet.  The  provision  for  two  types  of 
intercommand  RDT&E  support.  Fleet  and  Other,  permits  a controlled  method  for 
obtaining  the  requisite  support  through  a single  coordinating  agent. 

Issue  No.  5.  Retain  COMOPTEVFOR  involvement  in  the  T&E  of  training  devices  which 
Interface  directly  with  hardware  installed  on  operating  units. 

This  is  the  most  reliable  method  of  insuring  that  the  device  does  not  affect 
operational  capabilities. 

Issue  No.  6.  Define  OTE  limits  to  a measurement  of  training  device  effectiveness 
oriTy"  i n terms  of  course  behavioral  objectives  and  stipulated  course  requirements. 

Even  where  possible,  it  is  expensive  in  terms  of  time  and  resources  to 
determine  transfer  of  training  to  the  operational  setting.  OTE  results  will  be 
reported  only  with  respect  to  how  well  the  device  supports  its  training  program. 

Issue  No.  7.  There  is  a need  to  input  to  the  ASU  decision  making  process  a cost 
effectiveness  recommendation  for  training  devices.  An  agent  must  be  assigned 
this  responsibility. 

A training  device  evaluation  is  not  complete  until  a cost  comparison  is 
made,  where  feasible,  of  the  life  cycle  costs  of  existing  training  devices  and 
their  training  effectiveness  with  the  life  cycle  costs  of  the  new  device  and  its 
training  effectiveness.  The  results  should  be  one  input  to  the  decision  to 
acquire  additional  devices.  The  issue  generated  by  this  requirement  is  which 
agent  should  perform  the  evaluation,  the  DA,  the  OTA,  or  an  independent  agent. 

Issue  No.  8.  Convene  a conference  to  prepare  the  proposed  instruction  for  CNO 
release. 

Attendees  at  this  conference  should  include  representati ves  of  the  following 
commands  and  agencies:  CNO  (0P-01,  0P-02,  0P-03,  0P-04,  0P-05,  OP-096,  OP-098), 
Comptroller  of  the  Navy,  CNM,  CNET,  COMOPTEVFOR,  CNTECHTRA,  CNATRA,  TAEG,  and 
NAVTRAEQUIPCEN. 

Issue  No.  9.  Coordination  is  needed  to  relate  the  MILCON  budget  cycle  to  the 
device  acquisition  process. 

MILCON  funding  currently  requires  a 5-year  planning  cycle.  Training 
device  planning  seldom  extends  beyond  a 3-  to  4-year  period  for  the  prototype 
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and  1*.  to  1 years  tor  production  units.  This  timing  problem  has  not  been  resolved 
in  the  present  study. 
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PROPOSED  OPNAV  INSTRUCTION 

OPNAV  INSTRUCTION 

From:  Chief  of  Naval  Operations 
Subj:  Test  and  Evaluation  of  Training  Devices 

Ref:  (a)  D00  Directive  5000.1 

(b)  DOD  Directive  5000.3 

(c)  SECNAViNST  5000.1 

(d)  OPNAV INST  3960.10 

(e)  OPNAV INST  4720. 9D 

Enel:  (1)  Certification  of  Readiness  for  TRAEVAL 

(2)  Device  Test  and  Evaluation  Master  Plan  (DTEMP) 

(3)  Requests  for  RDT&E  Support 

(4)  Reports  Symbols 

1.  Purpose.  This  instruction: 

a.  Implements  policy  established  by  reference  (c)  within  the  Navy  for 
training  devices  acquired  with  RDT&E  funds. 

b.  Establishes  the  method  for  conducting  test  and  evaluation  (T&E)  in 
Navy  acquisition  programs  for  training  aids  and  devices. 

c.  Defines  the  T&E  responsibilities  of  CNO,  Director  RDT&E,  Resource  and 
Assessment  Sponsors,  CHNAVMAT,  Developing  Agencies  (DA),  Operational  Test 
Agencies  (OTA),  and  fleet  commanders. 

d.  Establishes  procedures  for  planning,  conducting,  and  reporting  T&E. 

e.  Establishes  the  relationship  between  development  T&E  (DTE)  and  opera- 
tional T&E  (OTE)  agents. 

f.  Establishes  procedures  and  format  for  Device  Test  and  Evaluation 
Master  Plans  (DTEMP). 

I 

g.  Establishes  procedures  for  obtaining  Fleet  or  Other  unit  RDT&E  Support 
for  R&D  that  is  not  part  of  an  acquisition  program. 

2.  Background.  Department  of  Defense  (DOD)  policy  for  defense  systems  acquisi- 
tion was  established  by  references  (a)  and  (b)  and  implemented  within  the  Navy  by 
reference  (c).  Reference  (d)  is  the  implementing  document  for  this  policy  as  it 
affects  operational  systems.  This  instruction  is  the  implementing  document  for 
Navy  training  device  acquisition  and  parallels  reference  (d)  insofar  as  possible. 

No  change  to  the  basic  DOD  policy  is  intended.  The  key  element  of  DOD  acquisi- 
tion policy  affecting  T&E  is  that  "Programs  shall  be  structured  and  resources 
allocated  to  ensure  that  the  actual  achievement  of  program  objectives  is  the 
pacing  function."  T&E  is  a principal  tool  used  to  demonstrate  the  achievement 

of  program  objectives.  T&E  is  mandatory  in  all  procurements  for  which  first 
item  acquisition  was  accomplished  using  research  funds. 
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3.  Scope  and  Applicability.  Ihis  instruction  applies  to  all  Navy  training 
device  acquisition  programs,  whether  they  are  independent  programs  or  are  related 
to  a specific  system  or  hardware,  except  nuclear  weapon  subsystems  or  nuclear 
propulsion  subsystems.  Nuclear  subsystems  are  governed  by  joint  DOD  and  ERDA 
agreements.  In  addition,  this  instruction  applies  to  programs  requiring  major 
modifications  to  training  devices. 

Training  devices  that  employ  new  or  off-the-shelf  computers  and  systems 
components,  but  have  training  system  unique  software  and  interface  components, 
will  be  developed  and  procured  with  RDT&E  funds.  The  initial  or  prototype 
system  and  all  of  its  support  costs  through  service  acceptance  will  be  funded  in 
RDT&E.  RDT&E  will  not  fund  beyond  the  initial  system  unless  more  than  one  full 
system  is  required  to  demonstrate  that  the  training  device  meets  all  effec- 
tiveness and  supportabil ity  objectives. 

4 . Definitions. 

a.  Training  device.  The  hardware  and  software  which  has  been  designed, 
or  modified,  exclusively  for  training  purposes,  and  which  usually  involves  to 
some  degree  simulation  or  stimulation  in  its  construction  or  operation,  so  as  to 
demonstrate  or  illustrate  a concept  or  simulate  an  operational  circumstance  or 
environment. 

b.  Training  effectiveness  is  the  relationship  of  the  improvement  demon- 
strated by  persons  subsequent  to  the  completion  of  a training  experience  compared 
to  the  capabilities  demonstrated  by  a similar  group  performing  to  identical 
criteria  who  have  not  had  the  training  experience.  This  relationship  can  be 
positive,  which  indicates  the  training  experience  was  effective,  or  it  can  be 
zero  or  negative,  which  indicates  the  training  experience  was  not  effective. 
Training  effectiveness  can  be  measured  objectively,  subjectively,  or  in  some 
combination.  For  a training  device,  student  performance  is  compared  to  the 
training  system's  stated  goals  for  the  device. 

c.  Major  Modification.  Any  change  in  design  in  hardware  or  software  to 
existing  training  device  or  operational  hardware  or  a system  which  alters 
substantially  the  operational,  training,  logistics,  or  other  characteristics. 

This  includes  reliability  and  maintainability  characteristics. 

d.  Service  acceptance.  The  situation  wherein  a ‘raining  device  has  been 
tested  and  documented  in  accordance  with  reference  (e),  and  has  been  certified 
as  approved,  or  provisionally  approved,  for  service  use. 

5.  Device  Acquisition  Categories,  four  acquisition  categories  (ACATs)  are 
established"  to  govern  acquisition  proceduies  lhese  ACATs  and  their  controlling 
documents  are  defined  in  reference  (d).  TdDle  1 is  a suninary  of  the  ACATs. 
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TABIE  1.  ACQUISITION  CATEGORIES 

\CAT  DESIGNATED  BY  NOMINAL  DOLLAR  VALUE  THRESHOLD  OTHER  CRITERIA 


RDTSE 

PRODUCTION 

I 

SECDEF  OR 
DEPSECDEF 

$S0M 

$200M 

Lesser  Programs 
designated  by 

SECDEF  or  DEPSECDEF . 

II 

DSARC  principal, 
SECNAV,  or  CNO 

S20M 

$ 50M 

Lesser  programs 
recommended  by 
CHNAVMAT , OP -090. 

OP -098,  or  program 
sponsor 
(DCNO/DMSO) . 

III* 

Program  Sponsor 

$ 5M 

$ 20M 

Lesser  programs 
recommended  by 
CHNAVMAT,  OP -090, 

OP -098,  or  DA. 

IV* 

CHNAVMAT/CNET 

Includes  all  pro- 
grams not  designated 
ACAT-I , II,  or  III. 

♦Programs  below  the  ACAT-1II  dollar  threshold  will  normally  be  designated  ACAT-III 
if  they: 


a.  directly  affect  the  military  characteristics  of  ships,  aircraft, 
or  other  combatant  units;  or 

b.  require  OTE  to  support  key  program  decisions;  or 

c.  require  RDT&E  support. 

The  majority  of  training  devices  fall  in  ACAT-III  or  IV.  Many  ACAT-IV  devices 
are  designated  as  ACAT-III  for  other  reasons.  Moreover,  not  all  device  acquisi 
tion  programs  require  the  same  level  of  attention,  or  the  same  degree  of  T&E. 
Therefore,  within  the  ACAT-III  and  ACAT-IV  categories,  three  device  categories 
(DCAT)  are  established  to  govern  the  acquisition  process,  to  include  T&E.  With 
respect  to  T&E,  all  training  device  acquisition  programs  require  a DTE,  whereas 
only  those  programs  designated  DCAT  A or  B require  an  independent  OTE.  Table 
2 is  a summary  of  the  proposed  DCATs. 
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TABLE  2.  DEVICE  CATEGORIES 


DCAT 

DESIGNATED  BY  NOMINAL  DOLLAR 

VALUE  THRESHOLDS 

CRITERIA 

RDT&E 

PRODUCTION 

A* 

Assessment  Sponsor 

$1 . 5M 

$ 5M 

Other  programs 
recommended  by 
CNET,  OP-098, 
or  DA. 

B 

Assessment  Sponsor 

Training  devices 
which  interface 
directly  with 
operational 
systems  and 
require  OTE  with 
that  system. 

C* 

CNET 

A1 1 programs  not 
designated  DCAT- 
A or  B. 

♦Programs  below  the  DCAT-A  dollar  threshold  will  normally  be  designated 
DCAT-A  if  they: 

a.  require  an  independent  OTE  to  support  key  program 
decisions,  or 

b.  require  Fleet  or  Other  RDT&E  Support. 

6.  Navy  T&E  Policy.  The  Navy  T&E  policy  is  the  same  as  that  of  reference  (b). 
This  instruction  does  not  repeat  this  policy,  thus  direct  reference  to  reference 
(b)  is  required. 

7.  Types  of  T&E.  There  are  two  types  of  T&E--DTE  and  OTE.  The  authority  for 
conducting  each  is  delegated  to  a different  organization.  Phasing  of  training 
device  T&E  is  shown  in  figure  1 and  described  in  subsequent  paragraphs. 


' 
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- 
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Program  Phase 


Type  T&E 
OTE 
OTE 


* Not  Required  for  Most  Training  Device  Acquisition  Programs 
**  Technical  Evaluation  (TECHEVAL) 

***  Training  Effectiveness  Evaluation  (TRAEVAL) 

Figure  1.  TXE  Phases 

a.  DTE.  DTE  is  defined  in  reference  (b).  The  DA  is  responsible  for 
DTE.  The  DA~for  training  devices  is  a CNET  designated  command,  a systems  command, 
or  a CHNAVMAT -designated  project  manager  assigned  the  responsibility  for  the 
total  acquisition  program.  The  DA  is  usually  designated  at  Milestone  0.  DTE  is 
planned  by,  conducted  by  or  for,  monitored  by,  and  reported  by  the  DA.  The  DA 
shall  establish  early  and  continuing  liaison  with  the  OTA  to  insure  that  the  DTE 
program  is  fully  understood  and  that  OTE  requirements  are  identified  and  integrated 
into  the  program  schedule  with  proper  budgeting.  The  DA  shall  provide  the  OTA 
with  all  significant  DTE  test  data  and  analyses  that  will  assist  in  the  planning 
and  interpreting  of  OTE.  DTE  test  data  indicating  failures  or  anomalies  will  be 
provided  as  rapidly  as  possible  to  the  Assessment  Sponsor  and  OTA. 

(1)  DTE  is  required  for  all  acquisition  progams  in  all  four  ACATs  and 
al 1 three  DCATs. 

(2)  DTE  is  conducted  in  four  major  phases.  The  specific  objectives 
of  each  phase  and  each  subphase  of  DTE  are  developed  by  the  DA  and  published  in 
the  DTEMP. 


(a)  DT-1  is  the  DTE  which  may  be  conducted  during  the  conceptual 
phase  to  support  the  program  initiation  decision.  Most  device  acquisition  pro- 
grams do  not  require  DT-I.  However,  when  required,  it  will  normally  consist  of 
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concept  evaluations  and  trade-off  analyses.  Design  risks  and  alternative  solu- 
tions are  identified  when  the  DT-I  is  conducted. 

(b)  DT-Il  is  that  DTE  conducted  during  the  validation  phase  to 
support  the  full  scale  development  (Engineering)  decision.  It  demonstrates  that 
the  design  risks  have  been  minimized  and  that  the  most  cost  and  training  effective 
solution  is  selected.  It  is  normally  conducted  at  the  subsystem/component 
level,  up  to  and  including  employment  of  engineering  model s/brassboards  for 

final  evaluation. 

(c)  DT-I 1 1 is  that  DTE  conducted  during  the  full  scale  develop- 
ment phase  to  support  the  first  major  production  decision.  The  first  major 
production  decision  is  that  first  decision  to  produce  systems  for  permanent 
installation  or  use,  or  for  inventory.  It  follows  approval,  or  provisional 
approval  for  service  use,  or  the  granting  of  a waiver  of  approval  for  service 
use.  DT-III  demonstrates  that  the  engineering  design  meets  performance,  reli- 
ability, maintainability,  supportabi 1 i ty,  environmental  compatibility,  and 

system  safety  requirements.  Subphases  of  DT-III  may  include  contractor  evaluations 
and  tests,  formal  Naval  acceptance  tests,  and  the  like.  The  final  phase  of  DT-III 
is  TECHEVAL,  the  purpose  of  which  is  to  certify  that  the  device  meets  specified 
technical  requirements  and  is  ready  for  a Training  Effectiveness  Evaluation 
(TRAEVAL).  Enclosure  (1)  contains  instructions  for  certification  of  readiness 
for  TRAEVAL. 


(d)  DT-IV  is  that  DTE  conducted  after  the  first  major  production 
decision  on  a production  unit  to  verify  that  product  improvement  or  correction 
of  design  deficiencies  discovered  during  prior  testing  or  operational  use  have 
been  effectively  completed. 

(3)  DTE  may  be  divided  into  subphases  (e.g.,  DT-IIIA,  DT-IIIB,  etc.), 
as  necessary. 

(4)  The  OTA  shall  monitor  all  pertinent  phases  of  DTE.  OTA  comments 
shall  be  included  with  all  reported  DTE  events. 

b.  Operational  Test  and  Evaluation  (OTE).  OTE  is  defined  in  reference  (b). 
The  Navy  is  required  to  have  one  organization,  separate  and  distinct  from  the 
developing  and  procuring  command  and  from  the  using  command,  which  will  be 
responsible  for  all  OTE.  For  operational  hardware  and  systems  this  agency  is 
Commander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFOR) . Because  of  the 
nature  of  training  devices,  the  special  character  of  their  use,  and,  consequently, 
the  special  concern  with  the  operational  testinq  to  which  they  are  subjected, 
the  Chief  of  Naval  Education  and  Training  (CNET)  is  designated  the  Navy  organiza- 
tion responsible  for  the  OTE  of  all  training  devices  which  require  an  independent 
OTE.  OTE  authority,  with  the  approval  of  OP-098,  will  be  designated  to  a command 
which  is  independent  of  the  developing,  procuring,  and  using  command.  Designation 
of  the  OTA  will  occur  at  the  same  time  as  the  designation  of  the  DA  to  facilitate 
coordination.  The  OTA  will  provide  the  DA  with  all  major  OTE  requirements,  test 
data,  and  analyses.  When  the  test  data  indicate  failures  or  anomalies,  they 
will  be  provided  as  rapidly  as  possible  to  the  Assessment  Sponsor  and  the  DA. 
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8.  Device  Test  and  Evaluation  Master  Plan  (DTEMP).  The  DTEMP  is  the  controlling 
management  document  for  training  device  T&E.  It  defines  the  test  and  evaluation 
required  for  training  device  acquisition  programs  acquired  with  RDT&E  funds, 
except  as  noted  below.  It  contains  the  integrated  testing  requirements  of  the 
DA  (for  DTE)  and  the  OTA  (for  OTE),  the  initial  schedule  of  events,  and  all 
resources  estimated  to  be  required  for  T&E  accomplishment. 

a.  CNO  approved  DTEMPs  are  required  for  all  ACAT-I,  II,  III  (DCAT-A  and 
B)  and  IV  (DCAT-A  and  B)  programs.  For  DCAT-C  training  device  acquisition 
programs,  CNET  will  promulgate  instructions  for  the  preparation  and  promulgation 
of  T&E  plans.  T&E  plans  for  DCAT-C  programs  will  not  be  referred  to  as  DTEMPs. 

b.  The  DTEMP  will  be  prepared  early  in  each  new  training  device  acqui- 
sition program  and  approved  by  CNO  (OP-098)  prior  to  Milestone  1.  The  initial 
version  of  the  DTEMP  will  lack  many  specifics.  Revisions  based  on  reviews  and 
development  of  technical  data  will  add  detail  as  developed. 

c.  The  DTEMP  will  be  prepared  by  the  DA  in  cooperation  with  the  OTA.  The 
OTE  portions  of  the  DTEMP  will  be  prepared  by  the  OTA.  The  DA  shall  insure  that 
the  DTEMP  accurately  reflects  the  planned  approach  to  provide  necessary  T&E  to 
solve  design  issues.  Enclosure  (2)  contains  instructions  for  DTEMP  preparation. 

d.  The  DTEMP  (and  major  revisions  thereto)  will  be  submitted  by  the  DA  to 
the  Assessment  Sponsor,  via  CNET  and  OP-098,  for  approval.  Where  complete 
agreement  between  the  DA  and  the  OTA  on  the  proposed  plan  cannot  be  obtained 
prior  to  submission  to  CNET,  the  DA  will  state,  in  writing,  the  areas  of  disagree- 
ment and  his  reasons  therefore,  with  a copy  to  the  OTA.  The  OTA  shall  provide, 

in  writing,  the  rationale  for  the  need  to  accomplish  the  tests,  to  follow  the 
described  procedures,  and/or  the  need  for  stipulated  resources  that  appear  to  be 
in  disagreement  with  the  DA's  planned  approach.  In  order  to  insure  standard 
format  and  procedures,  OP-098  will  draft  and  staff  any  CNO  revisions  of  DTEMPs. 

In  case  of  disagreement  between  CNET,  OP-098,  and  the  Acquisition/Resource 
Sponsor,  resolution  will  be  requested  of  0P-090  and  the  VCNO,  in  that  order. 

e.  Approval  of  the  DTEMP,  or  revisions  thereto,  constitutes  CNO  direction 
to  conduct  the  T&E  program  defined  therein,  including  the  commitment  of  RDT&E 
Support,  other  command  support,  and  the  expenditure  of  resources.  Test  plans 
will  be  drawn  up  directly  from  the  DTEMP  by  the  DA  for  DTE  and  production 
acceptance  test  and  evaluation,  and  by  the  OTA  for  OTE.  These  test  plans  will 
be  consistent  with  the  DTEMP  and  adequate  to  carry  out  its  provisions. 

f.  The  DTEMP  will  be  reviewed  by  the  DA  and  OTA  as  often  as  necessary, 
but  not  less  than  annually  and  about  2 months  prior  to  each  major  decision 
milestone.  Based  on  these  reviews,  the  DTEMP  will  be  updated  or  revised,  as 
necessary,  to  incorporate  significant  results  achieved  and  changes  to  plans  and 
milestones.  The  reasons  for  all  changes  will  be  documented. 

g.  Minor  changes  to  the  DTEMP  may  be  made  by  the  DA  (for  DTE)  or  the  OTA 
(for  OTE)  without  CNO  approval.  RDT&E  Support  . .anges  are  never  considered 
minor.  For  minor  changes  the  DA  or  OTA  shall  advise  the  Assessment  Sponsor  in 
writing  of  the  change  and  the  rationale  as  to  why  it  is  required.  A copy  of 
this  letter  shall  be  provided  to  OP-098,  CNET,  the  Resource  Sponsor,  and  to  the 
OTA  or  DA. 
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h.  DTEMPs  for  Navy  led  joint  service  developments  and  training  device 
acquisition  programs  will  be  prepared  in  active  and  close  coordination  with  the 
other  participating  service(s).  They  will  be  approved  jointly  by  CNO  and  the 
military  chief(s)  of  the  other  participating  service(s). 

i.  The  DTEMP  shall  be  retired  upon  the  completion  of  the  last  phase  of 

OTE. 

9.  Special  T&E  Situations.  There  are  several  types  of  programs  or  occurrences 
in  which  special  T&E  situations  may  exist.  These  programs  are  discussed  in  the 
following  paragraphs. 

a.  OPTEVFOR  Involvement.  As  shown  in  table  2,  training  device  acquisi- 
tion programs  designated  ACAT-III  and  IV,  DCAT-B,  are  those  which  interface 
directly  with  operational  systems  or  hardware  and,  therefore,  could  affect  the 
operational  capabilities  of  the  equipment.  These  devices  must  be  tested  opera- 
tionally with  the  operational  system  to  insure  their  operational  suitability. 

The  OTA  for  these  training  devices  shall  be  COMOPTEVFOR.  Testing  management 
procedures  as  stipulated  in  this  instruction  shall  apply.  When  requested, 
educational  and  training  expertise  shall  be  provided  to  COMOPTEVFOR  by  CNET. 

b.  Combined  Testing.  The  policy  established  in  reference  (b)  will  be 
adhered  to  in  combined  development  and  operational  testing. 

c.  Test  and  Evaluation  Coordinating  Group  (TECG).  Complex,  many-faceted 
programs  may  require  extensive  T&E  coordination.  To  facilitate  this,  a TECG  may 
be  established  Dy  the  Assessment  Sponsor.  Membership  in  the  TECG  should  include 
the  Assessment  Sponsor,  Resources  Sponsor,  OP-102,  OP-983,  CNET,  DA,  OTA,  fleet 
representatives,  and  others,  as  appropriate.  The  TECG  shall  meet  at  the  direction 
of  the  chairman  who  shall  be  the  Assessment  Sponsor.  TECG  recommendations  will 

be  considered  for  inclusion  in  tne  DTEMP. 

d.  Deficiency  Reports.  If,  for  any  reason,  OTE  is  unlikely  to  be  success- 
fully or  efficiently  prosecuted,  the  OTA  will  transmit  by  message  a "Deficiency 
Report"  to  CNO,  CNET,  and  the  DA,  and  will  suspend  OTE.  This  will  be  followed 

by  a letter  report  which  outlines  the  problem,  a proposed  solution,  and  a pro- 
posed revised  schedule  for  OTE. 

e.  OTE  Support  for  USMC.  When  the  Commandant  of  the  Marine  Corps  (CMC) 
desires  OTE  support  for  training  devices,  he  will  request  such  support  from  CNO 
who  will  direct  CNET  to  provide  the  approved  support.  All  such  OTE  accomplished 
will  be  planned,  conducted,  and  reported  in  accordance  with  this  instruction, 
except  that  OTE  planning  will  be  coordinated  with  CMC,  and  the  OTA  will  report 
his  independent  tests  and  evaluations  to  both  CNO  and  CMC. 

f.  Joint  Test  and  Evaluation  (JTE).  JTE  is  that  T&E  conducted  by  two  or 
more  participating  services.  There  are  two  types  of  JTE: 

(1)  Service  initiated  JTE  in  joint  development  and  acquisition  programs. 
All  such  JTE  of  training  devices  for  which  the  Navy  is  the  lead  service  will  be 
planned,  accomplished,  and  reported  in  accordance  with  reference  (b)  and  this 
instruction,  unless  otherwise  directed. 
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(2)  Deputy  Director,  Test  and  Evaluation  (DDT&E)  initiated  JIL  is 
initiated  and  coordinated  by  DDT&E,  with  eventual  specific  delegation  to  one  of 
the  services  of  all  practical  aspects  of  the  JTE.  Prior  to  assignment  to  a 
service,  OP-098  and  CNET  will  serve  as  Navy  points  of  contact  for  DDT&E  on  train- 
ing device  JTE  matters. 

g.  Wa ivers  of  T&E. 

(1)  Any  waiver  of  the  accomplishment  of  T&E  outlined  in  the  Decision 
Coordinating  Paper  ( DCP ) for  an  ACAT-I  program  will  be  granted  only  by  SECDEF. 

(2)  Any  waiver  of  the  accomplishment  of  T&E  outlined  in  the  Program 
Memorandum/Navy  DCP  (PM/NDCP)  for  an  ACAT-11  program  will  be  granted  only  by 
SECNAV,  Under  SECNAV,  or  an  ASN  designated  by  SECNAV. 

(3)  Any  waiver  of  accomplishment  of  other  T&E  outlined  in  an  approved 
DTEMP  will  be  granted  only  by  CNO  for  ACAT-1  programs  and  by  OP-098  for  ACAT-I I 
and  111  programs. 

(4)  In  the  event  T&E  would  cause  a delay  in  obtaining  required  training 
devices  such  that  operational  capabilities  could  be  degraded,  a request  for  a 
waiver  of  the  T&E  requirement  prior  to  the  obligation  of  funds  for  obtaining 
production  units  will  be  submitted  to  OP-098  via  CNET  and  the  Assessment  Sponsor. 
The  request  shall  contain  the  reason  the  waiver  is  needed  in  terms  of  how  it 

will  impact  operational  capabilities  and  the  number  of  units  required.  Granting 
of  the  waiver  does  not  eliminate  the  requirement  for  T&E.  Rather  it  authorizes 
the  acquisition  of  a limited  number  of  devices  pending  the  outcome  of  T&E. 

10.  RDT&E  Support.  There  are  two  categories  of  RDT&E  Support  available  to  the 
test  agenciesT  Fleet  RDT&E  Support  is  that  support  provided  by  fleet  operational 
units  and  personnel.  Other  RDT&E  Support  is  any  support  provided  by  units  or 
personnel  not  under  an  operational  commander. 

a.  Fleet  RDT&E  Support.  This  term  encompasses  the  operating  and  non- 
operating support  provided:  b^  operational  naval  forces  having  a primary 
mission  other  than  R&D;  to  the  DA,  OTA,  or  an  R&D  agency;  for  the  accomplishment 
of  acquisition  program  T&E,  or  research  and  development  not  related  to  specific 
acquisition  programs.  There  are  three  types  of  Fleet  RDT&E  Support:  dedicated 
support  precludes  employment  of  the  supporting  unit  in  other  missions;  con- 
current support  permits  employment  of  the  supporting  unit  in  activities  other 
than  RDT&E  Support,  but  will  have  an  operational  impact  upon  the  unit's  employ- 
ment; and  NIjl  (not-to-interfere-basis)  support  permits  employment  of  the  supporting 
unit  without  interference  from  the  RDT&E  effort. 

(1)  Fleet  RDT&E  Support  requirements  are  compiled  from  three  inputs: 

(a)  Approved  Test  and  Evaluation  Master  Plans  (TEMP)  for  ACAT-I, 

1 1 , and  1 1 1 programs . 

(b)  Approved  DTEMPs  for  ACAT-I  and  II,  and  ACAT  ill  and  V 
(DCAT-A  and  B)  programs. 
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(c)  Requests  for  Fleet  RDT&E  Support,  for  R&O  not  related  to 
specific  acquisition  programs  will  be  submitted  to  CNO  for  approval  by  the  R&D 
agency.  See  enclosure  (3)  for  instructions. 

(2)  From  these  three  inputs,  CNO  (OP-098)  will  compile  and  publish, 
annually,  "CNO  Long-Range  Fleet  RDT&F  Support  Requirements"  for  the  budget-  and 
out-years.  Fleet  commanders  will  use  this  report  for  guidance  in  planning, 
programming,  and  budgeting  for  Fleet  RDT&E  Support. 

(3)  Using  these  same  inputs,  updated  by  conf irmation  procedures,  CNO 
(OP-098)  will  compile  and  publish,  quarterly,  "CNO  Quarterly  Fleet  RDT&E  Support 
Requirements"  for  the  forthcoming  quarter.  This  summary  will  be  used  at  quarterly 
fleet  scheduling  conferences  to  establish  the  requirements  for  Fleet  RDT&E 
Support. 


(4)  CNO  (0P-098)  will  assign  a priority  (applying  to  fleet  support 
only)  to  each  Fleet  RDT&E  Support  task  listed  in  the  CNO  Quarterly  Fleet  RDT&E 
Support  Requirements. 

(a)  Priority  ONE  support  tasks  take  precedence  over  normal  fleet 

operations. 

(b)  Priority  TWO  support  tasks  take  precedence  with  normal  fleet 

operations. 

(c)  Priority  THREE  support  tasks  take  precedence  after  normal 
fleet  operations. 

The  determining  factor  in  the  assignment  of  priorities  will  be  the  agency  of 
maintaining  the  RDT&E  schedule.  The  priorities  do  not  necessarily  have  a direct 
relationship  to  the  importance  of  the  program  supported.  The  type  of  support 
(dedicated,  concurrent,  or  NIB)  is  not  considered  in  determining  priority. 

(5)  Fleet  commanders  in  chief  will  schedule  support  tasks  listed  in 
the  CNO  Quarterly  Fleet  RDT&E  Support  Requirements  in  accordance  with  assigned 
priorities.  The  OTA  will  coordinate  Fleet  RDT&E  Support  scheduling  for  CNO  and 
will  report  to  CNO,  quarterly,  the  Fleet  RDT&E  Support  provided. 

(6)  Fleet  commanders  in  chief  are  requested  to  provide  any  assistance 
required  by  the  OTA  in  the  prosecution  of  CNO-approved  programs. 

(7)  Fleet  RDT&E  Support  will  not  be  provided  except  in  accordance 
with  the  provisions  of  this  instruction. 

b.  Other  RDT&E  Support.  This  term  encompasses  the  operating  and  non- 
operating support  provided:  b^  any  command,  activity,  or  agent  not  under  the 
direct  operational  conmand  of  a fleet  commander  and  having  a primary  mission 
other  than  R&D;  to  the  DA,  OTA,  or  an  R&D  agency;  for  the  accomplishment  of 
acquisition  program  T&E,  or  research  and  development  not  related  to  a specific 
training  device  acquisition  program.  The  three  types  of  R&D  support;  i.e., 
dedicated,  concurrent,  and  NIB,  are  applicable  to  Other  RDT&E  Support. 
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(1)  Other  ROr&L  Support  requirements  are  compiled  from  two  inputs: 

(a)  Approved  DTEMPs  for  ACAT-I  and  11  and  ACAT-III  and  IV  (DCAT- 
A and  B)  programs. 

(b)  Requests  for  Other  RDT&E  Support  for  R&D  not  related  to 
specific  training  device  acquisition  programs  which  have  been  approved  by  CNO. 

See  enclosure  (3)  for  instructions. 

(2)  From  these  two  inputs,  CNO  (OP-098)  will  compile  and  publish, 
annually,  the  "CNO  Long-Range  Other  RDT&E  Support  Requirments"  for  the  budget- 
and  out-years.  This  report  will  be  used  for  guidance  in  planning,  programming, 
and  budgeting  for  Other  RDT&E  Support. 

(3)  CNO  (OP-098)  will  assign  a priority  to  each  Other  RDT&E  Support 
task  listed  in  the  CNO  Long-Range  Other  RDT&E  Support  Requirements. 

(a)  Priority  ONE  support  tasks  take  precedence  over  normal 

functions. 

(b)  Priority  TWO  support  tasks  take  precedence  with  normal 

functions. 

(c)  Priority  THREE  support  tasks  take  precedence  after  normal 

functions. 

The  determining  factor  in  the  assignment  of  priorities  will  be  the  urgency 
of  maintaining  the  R&D  schedule.  The  priorities  do  not  necessarily  have  a 
direct  relationship  to  the  importance  of  the  program  supported.  The  type  of 
support  (dedicated,  concurrent,  or  NIB)  is  not  considered  in  assigning  priorities. 

(4)  Support  tasks  will  be  scheduled  as  listed  in  the  CNO  Long-Range 
Other  RDT&E  Support  Requirements  in  accordance  with  assigned  priorities.  For 
training  device  acquisition  programs,  the  OTA  will  coordinate  Other  RDT&E  Support 
scheduling  for  CNO.  In  R&D  programs  which  are  not  related  to  a specific  acqui- 
sition, the  DA  will  coordinate  Other  RDT&E  Support  scheduling  for  CNO.  A quarterly 
report  will  be  submitted  to  CNO  of  Other  RDT&E  Support  provided.  Negative 
reports  are  not  required. 

1 1 . T&E  Funding  Responsibility. 

a.  The  DA  will  plan,  program,  budget,  and  fund  the  cost  of  all  resources 
identified  in  the  approved  DTEMP  (or  the  approved  T&E  Plan  for  all  programs 
which  do  not  require  a DTEMP)  for  all  T&E  through  DT-IV  and  OT-IV,  except  fleet 
travel  and  operating  costs  for  Fleet  RDT&E  Support,  agency  travel  and  normal 
operating  costs  for  Other  RDT&E  Support,  and  OTA  travel  and  non-program  related 
administrative  costs.  The  DAs  funding  responsibilities  include  the  following. 

(1)  All  DTE  costs. 

(2)  All  OTE  costs  through  OT-IV,  including  laboratory  and  contract 
analytic  support,  instrumentation,  data  collection  and  reduction  (including  ADP 
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services  and  administration),  establishment  of  training  and  training  costs, 
expendables,  maintenance  and  logistic  support,  test  articles,  OTA  program  related 
costs,  etc.,  except  fleet,  agency,  and  OTA  costs  noted  in  subparagraph  a.  above. 
Since  the  DA  must  budget  for  these  costs,  it  is  essential  that  the  OTA  identify 
all  resource  requirements  in  sufficient  time  to  integrate  these,  through  the 
DTEMP,  into  the  program  schedule  and  the  budget  cycle. 

b.  Fleet  Commanders  will  plan,  program,  budget,  and  fund  fleet  travel 
costs  and  operating  costs  for  Fleet  RDT&E  Support  (including  A1RTEVR0N  aircraft 
operating  costs)  and  all  costs  of  OT-V  except  the  procurement  cost  of  the  training 
device  being  tested  and  OTA  travel  costs.  The  CNO  Long-Range/Quarterly  Flpet 
RDT&E  Support  Requirements  will  provide  requisite  guidance. 

c.  Agencies  designated  to  provide  Other  RDT&E  Support  will  plan,  program, 
budget,  and  fund  agency  travel  and  operating  costs  for  Other  RDT&E  Support.  The 
training  agent  will  budget  all  costs  of  OT-V  except  the  procurement  cost  of  the 
training  device  being  tested  and  OTA  travel  costs.  The  CNO  Long  Range  Other 
RDT&E  Support  Requirements  will  provide  requisite  guidance. 

d.  CNET  will  plan,  budget,  and  fund  the  OTA  travel  costs  and  nonprogram 
related  administrative  costs,  except  that  for  DCAT-B  OTE  programs,  COMOPTEVFOR 
will  assume  this  responsibility. 

e.  Responsibilities  for  T&E  costs  of  research  and  development  efforts  not 
related  to  specific  acquisition  programs  are  the  same  as  those  above,  except  the 
R&D  agency  has  responsibilities  equivalent  to  those  of  the  DA. 

12.  T&E  Identification.  To  assist  in  identifying  and  tracking  T&E  and  RDT&E 
Support  scheduling  and  accounting,  CNO  (OP-098)  will  assign  a T&E  number  to  each 
acquisition  program,  and  to  each  nonprogram-related  research  or  development 
effort  requiring  RDT&E  Support.  The  assigned  number  will  continue  for  the  life 
of  the  program.  For  ACAT-I  and  II  and  ACAT-III  and  IV  (DCAT-A  and  B)  programs, 
this  number  will  identify  the  DTEMP.  It  will  be  used  by  all  activities,  in 
combination  with  the  DT/OT  phase  number,  to  make  reference  to  T&E  on  the  specified 
system. 

13.  OPNAV  Focal  Point  for  T&E.  The  Director,  Test  and  Evaluation  Division  (OP- 
983)  is  the  focal  point  established  in  OPNAV,  in  accordance  with  reference  (b), 
to  assist  the  DA  and  OTA,  and  to  keep  the  CNO  fully  informed  of  evaluation 
requirements  and  results.  OP-983  is  the  Navy's  single  point  of  contact  with 
RDT&E  in  the  office  of  the  Secretary  of  Defense. 

14.  Program  Reviews.  A principal  purpose  of  T&E  is  to  assist  decision  makers 
at  key  milestones.  The  three  basic  program  milestones  of  every  Navy  acquisition 
program,  regardless  of  ACAT,  are  program  initiation,  full-scale  development,  and 
production.  Other  milestones,  keyed  especially  to  the  commitment  of  resources, 
may  be  appropriate  in  particular  programs  and  may  be  scheduled  by  the  Assessment 
Sponsor  or  the  Resource  Sponsor.  At  each  milestone  the  decision  authority  uses 
T&E  results  and  other  input  data  to  decide  whether  to  commit  added  resources  to 
the  program  thus  controlling  the  program  through  the  acquisition  process.  For 
ACAT-I,  II,  and  III  acquisitions,  the  program  review  board  provides  the  forum  at 
which  the  DTE  Advisor  and  the  OTE  Advisor  present  the  results  of  T&E.  For 
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ACAT-IV  training  device  acquisition  programs,  CNET  will  establish  comparable 
review  procedures. 

15.  Approval  for  Service  Use  (ASU).  ASU,  or  Provisional  ASU  (PASU),  is  a 
separate  determination  addressed  during  the  program  review  for  the  production 
decision  (Milestone  III).  ASU  and  PASU  procedures  are  governed  by  reference 
(e).  For  ACAT-I,  II,  and  III  training  device  programs,  the  program  decision 
authority,  program  review  board,  DTE  Advisor,  and  OTE  Advisor  are  those  shown  in 
figure  2.  ASU  and  PASU  procedures  for  ACAT-IV  acquisition  programs  for  training 
devices  will  be  established  by  CNET. 
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Figure  2.  Program  Review  Responsibilities  for  Training  Devices 


16.  Implementing  Actions.  T&E  of  training  devices  will  be  accomplished  in 
accordance  with  the  policy,  principles,  and  direction  contained  in  this 
instruction. 
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u.  Programs  lor-  which  the  first  major  production  decision  (Milestone  III) 
is  planned  within  6 months  of  the  date  of  this  instruction  are  excluded  from  the 
requirements  of  this  instruction. 

b.  For  other  existing  ACAT-I,  II,  and  III  programs,  the  DA  will  prepare  a 
DTEMP  (or  revise  the  existing  TEMP)  in  accordance  with  this  instruction  and 
forward  it  for  approval. 

(1)  The  schedule  for  submission  of  the  DTEMP  is: 

(a)  within  1 year  of  the  date  of  the  current  TEMP,  or 

(b)  at  least  2 months  prior  to  the  next  milestone  decision,  or 

(c)  within  6 months  from  the  date  of  this  instruction,  whichever 

occurs  first. 

c.  Existing  instructions  which  conflict  with  this  instruction  will  be 
revised  as  soon  as  possible,  but  not  less  than  6 months  from  the  date  of  this 
instruction. 

d.  CNET  will : 

(1)  Implement  this  instruction  within  the  Naval  Education  and  Training 
Command  (NAVEDTRACOM)  as  soon  as  possible,  but  not  later  than  6 months  from  the 
date  of  this  instruction. 

(2)  Recommend  an  OTA  within  the  NAVEDTRACOM  to  OP-098  as  soon  as 
possible  so  that  OTE  planning  and  procedures  can  be  implemented  within  the 
constraints  of  paragraph  16. b.  above. 

(3)  Coordinate  with  CHNAVMAT  the  transfer  of  management  responsi- 
bilities from  the  structure  as  defined  in  reference  (d)  to  the  management  structure 
as  established  for  training  devices  in  this  instruction.  Transfer  should  be 
completed  within  6 months  of  the  date  of  this  instruction. 

(4)  Submit  to  CNO  (OP-098),  within  3 months,  a listing  of  training 
device  acquisition  programs  in  ACATs  I,  II,  and  III,  in  existence  and  planned, 
which  are  covered  by  the  provisions  of  this  instruction.  Identify,  in  the 
listing,  those  programs  which,  in  CNET's  judgment,  should  be  upgraded  to  higher 
ACATs  under  the  criteria  shown  in  table  1.  For  applicable  acquisitions,  recommend 
appropriate  DCATs. 

e.  CHNAVMAT  will: 

(1)  Implement  this  instruction  within  NAVMAT  as  soon  as  possible,  but 
not  later  than  6 months  from  the  date  of  this  instruction. 

(2)  Coordinate  with  CNET  the  transfer  of  management  responsibilities 
from  the  structure  as  defined  in  reference  (d)  to  the  management  structure  as 
established  for  training  devices  in  this  instruction.  Transfer  should  be  com- 
plete within  6 months  of  the  date  of  this  instruction. 
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t.  CNt I , the  UA,  01 A , and  fleet  commanders  will  assume  funding  respon- 
sibilities outlined  in  this  instruction  immediately.  In  those  cases  where  this 
transfer  of  funding  responsibilities  introduces  requirements  for  unprogrammed 
funds,  this  question  will  be  addressed  in  the  same  manner  as  any  unfunded 
deficiency.  Where  the  deficiency  results  from  the  moving  of  a responsibility 
from  one  organization  to  another,  the  deficiency  will  be  addressed  to  CNO. 

17.  T&E  Reports.  T&E  reporting  will  be  correlated  to  key  decision  points  and 
secondary  milestones  as  established  by  sponsors.  Requirements  for  test  reports 
and  evaluation  reports  (DTE  and  OTE)  will  be  specified  in  the  appropriate  DTEMP. 
Reports  required  by  this  instruction  will  carry  the  OPNAV  report  symbols  and 
notation  specified  in  enclosure  (4). 
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CERTIFICATION  OT  READINESS  FOR  TRAEVAL 

1.  After  completion  of  TECHEVAL,  and  when  the  DA  judges  the  system  to  be  ready 
for  TRAEVAL,  he  will  certify  the  system  to  be  ready  for  TRAEVAL  to  CNO  (OP-098). 

The  report  will  be  made  by  letter  or  message,  information  to  Assessment  Sponsor, 
Resource  Sponsor,  CNET,  OTA,  fleet  and  school  comnands  involved,  and  other 
interested  commands.  The  report  will  address  the  certi fication  criteria  set 
forth  in  paragraph  2.  below,  and  will  either  certify  full  compliance  or  request 
waivers  with  justification  for  minor  items. 

2.  The  criteria  for  full  Certification  of  Readiness  for  TRAEVAL  are: 

a.  All  DTEMP  specified  DT-III  items  have  been  completed,  or  will  be 
completed  at  the  same  time  as  agreed  upon  TRAEVAL  items. 

b.  All  DTEMP  specified  objectives,  performance  thresholds,  etc.,  have 
been  met,  or  are  anticipated  to  be  met,  for  those  items  to  be  tested  simultan- 
eously with  TRAEVAL  items. 

c.  The  results  of  DTE  demonstrate  that:  (1)  engineering  is  reasonably 
complete,  (2)  all  significant  design  problems  (including  compatibility,  inter- 
operability, reliability,  maintainability,  and  logistical  supportabi 1 i ty)  have 
been  identified,  (3)  solutions  to  the  above  problems  are  in  hand,  and  (4)  the 
system  is  functioning  in  a technically  acceptable  manner. 

d.  There  is  a high  probability  that  the  system  will  perform  successfully 
in  TRAEVAL  and  will  meet  the  technical  and  operational  criteria  for  full  ASU  on 
completion  of  TRAEVAL. 

e.  Approved  system  operating  and  maintenance  documents,  including  3-M 
documentation,  have  been  distributed  for  TRAEVAL.  Reading  grade  levels  established 
by  the  DA  will  have  been  verified  using  Manuals,  Technical:  General  Style  and 
Format  Requirements  (MIL-M-38784A) . 

f.  The  system  Integrated  Logistics  Support  Plan  has  been  provided  to  the 

OTA. 

g.  Adequate  logistic  support,  including  spares  and  repair  parts,  ground 
support  equipment,  etc.,  are  available  for  TRAEVAL. 

h.  The  TRAEVAL  manning  of  the  system  is  the  same  (in  numbers,  rates, 
ratings,  and  experience  level)  as  is  planned  for  operational  units  under  normal 
operating  conditions. 

i.  The  Navy  Training  Plan  has  been  provided  to  the  OTA. 

j.  All  required  training  for  personnel  who  will  operate  and  maintain  the 
system  during  TRAEVAL  (including  OTA  personnel)  has  been  completed.  This  training 
is  representative  of  that  planned  for  follow-on  training. 
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k.  All  resources  required  for  TRAEVAL  (instrumentation,  targets,  expend- 
ables, etc.)  have  been  arranged  for  and  are  available. 

l.  The  system  provided  for  TRAEVAL  has  the  same  configuration  as  the 
expected  production  system.  (Note:  If  this  is  not  the  case,  specify  in  detail 
the  production  configuration,  and  state  differences.) 

3.  If  the  criteria  for  full  Certification  of  Readiness  for  TRAEVAL  have  not 
been  met,  the  report  must  offer  alternative  courses  of  action,  including  delaying 
TRAEVAL,  until  the  criteria  are  essentially  met,  giving  the  impact  of  each 
alternative,  and  the  rationale  for  the  recommended  course  of  action. 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  (DTEMP) 

1.  DTEMP  Concept.  The  DTEMP  is  a short,  concise  master  plan  for  training 
device  T&E.  The  initial  DTEMP  should  be  prepared  and  submitted  as  early  in  the 
acquisition  program  as  possible,  preferably  prior  to  Milestone  I.  It  should  be 
updated  whenever  major  changes  occur  and  reviewed  no  less  frequently  than  once 
per  year  or  2 months  prior  to  each  major  milestone.  Every  effort  should  be  made 
to  limit  the  DTEMP  to  20  pages  for  complex  programs  and  fewer  for  more  straight- 
forward ones.  The  DTEMP  is  the  single  management  document  which  integrates  the 
entire  T&E  effort  of  the  acquisition  program.  Its  purposes  are  to  direct  and 
control  the  accomplishment  of  adequate  T&E;  to  identify  all  required  T&E  resources 
to  facilitate  long  range  planning,  programming,  and  budgeting;  to  eliminate 
redundant  testing;  and  to  reduce  RDT&E  Support  to  the  essential  minimum.  It 

must  be  factual  and  specific,  avoiding  generalities,  and  emphasize  quantative 
values  whenever  possible.  The  DTEMP  forms  the  basic  contract  between  the  DA 
and  the  OTA  for  conduct  of  the  overall  T&E  effort. 

2.  DTEMP  Outline.  The  DTEMP  contains  7 parts: 

I.  Administrative  Information 

II.  Description 

III.  Integrated  Schedule 

IV.  DTE  Outl ine 

V.  OTE  Outline 

VI.  Resources  Summary 

VII.  References 

3.  DTEMP  Preparation.  In  T&E  planning,  which  is  the  core  of  the  DTEMP  prepara- 
tion, the  DA  is  the  authority  for  DTE  and  the  OTA  is  the  authority  for  OTE. 
Planning  must  be  done  in  active  and  close  coordination  between  the  DA  and  the 
OTA.  Each  principal  is  to  recommend  appropriate  changes  in  the  other's  plans 
and  to  be  receptive  to  change  recommendations  from  the  other  principal,  to  the 
end  that  adequate  T&E  is  accomplished  with  minimum  expenditure  of  effort  and 
resources.  Where  appropriate,  combined  DT/OT  will  be  planned,  provided  each 
principal  conducts  and  reports  their  portion  of  the  T&E  independently.  In 
preparing  the  DTEMP: 

a.  The  DA  drafts  Parts  I,  II,  IV,  and  VII. 

b.  The  OTA  drafts  Part  V. 

c.  The  DA  and  the  OTA  integrate  the  T&E  Outlines  (Parts  IV  and  V)  to 
produce  Parts  III  and  VI. 

d.  The  DA  submits  the  DTEMP  to  the  Assessment  Sponsor  for  approval  out- 
lining any  unresolved  issues  in  an  accompanying  memorandum. 

e.  The  OTA  submits  his  views  on  unresolved  issues  to  the  Assessment 
Sponsor  by  separate  memorandum,  copy  to  the  DA. 
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4.  DTEMP  Format.  TAB  A contains  the  standard  DTEMP  format  to  be  used  as  a 
guided  The  DTEMP  number  is  the  same  as  the  T&E  identification  number  in  the  CNO 
Index  of  Acquisition  Programs.  The  date  at  the  top  of  each  DTEMP  page  is  the 
date  that  page  was  last  revised.  The  following  comments  are  keyed  to  related 
sections  of  Tab  A. 

a.  Part  I.  Administrative  Information.  Keep  brief.  Include  only  essential 
information. 

b.  Part  II.  Description.  Each  section  should  include  the  following  infor- 
mation. If  any  section  is  inappropriate  to  the  training  device  under  consideration, 
a statement  to  that  effect  should  be  made  under  the  appropriate  heading. 

(1)  System  Description  and  Mission.  This  section  should  contain  a 
short  description  of  the  operational  system  and  the  training  system.  Include  a 
brief  rationale  for  the  needed  training  device.  The  training  system  and  device 
description  should  include  the  official  nomenclature. 

(2)  Critical  T&E  Issues.  The  critical  issues  in  the  device  acquisition 
documentation  should  be  reviewed  and  those  pertaining  to  the  T&E  program  should 

be  described  in  this  section.  Include  how  T&E  will  be  used  to  resolve  each 
issue. 


(3)  Objectives  and  Thresholds.  Overall  program  test  objectives  and 
thresholds  should  be  included  in  this  section  keyed  to  major  program  milestones 
and,  where  applicable,  to  major  operational  hardware  milestones. 

(4)  Required  Technical  Characteristics.  A list  of  key  technical 
characteristics  of  the  device  should  be  listed  in  this  section  showing  the  per- 
formance variables,  goals,  and  thresholds. 

(5)  Required  Operational  Characteristics.  Same  as  (4)  above,  expressed 
in  terms  of  training  effectiveness  and  operational  suitability. 

(6)  Environmental  Impact  Assessment  of  T&E.  Keep  brief.  The  impact 
of  the  device  in  terms  of  air,  water,  and  noise  pollution  is  to  be  discussed. 


c.  Part  III.  Integrated  Schedule.  This  part  will  consist  of  one  page 
which  may  be  a foldout.  It  displays  the  integrated  time  sequencing  of  test  and 
evaluation  to  include  DTE  (including  contractor  test  and  evaluation.  Navy 
preliminary  and  technical  evaluations,  acceptance  testing,  etc.),  OTE  (both  IOTE 
and  FOTE),  and  related  key  events  in  the  acquisition  decision-making  process.  A 
legend  may  be  used  for  essential  explanatory  notes.  However,  more  complete 
information  about  the  events  or  the  schedule  is  contained  in  the  DTE  and  OTE 
Outline  (Parts  IV  and  V).  The  following  typical  T&E  events  should  be  included 
in  the  integrated  schedule. 


(1)  Program  Milestones.  Both  device  and  operational  hardware  (if 
appropriate)  milestones  0,  I,  II,  and  III  are  to  be  depicted.  Device  ASU  and 
other  program  reviews  are  to  be  included. 
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(2)  Pertinent  T&E  data,  to  include  all  Navy  and  contractor  tests  and 
evaluations. 

(3)  Major  resource  availability  requirements. 

(4)  Key  dates  for  the  issuance  of  test  plans,  reports,  and  similar 

documents . 

d.  Part  IV . DTE  Qutl ine.  This  part  should  contain  all  planned  DTE  in 
sufficient  detail  that  resources  can  be  identified,  and  the  DA  can,  subsequently, 
develop  detailed  test  plans.  Although  the  near-term  events  defined  in  the 
outline  will  contain  fairly  precise  data,  the  long-range  portion  should  also  be 
as  complete  and  specific  as  possible  as  regards  schedules  and  resources. 

Security  of  equipment  and  operations  should  be  explicitly  covered  in  all  T&E 
planning.  The  DTE  Outline  will  contain  the  following  three  sections: 

(1)  DTE  to  Date.  This  section  will  contain  a summary  of  the  DTE 
conducted  prior  to  the  date  of  the  current  revision  to  the  DTEMP.  A brief 
description  of  the  actual  test  articles  (brassboard,  advanced  development  model, 
etc.)  with  emphasis  on  how  the  operational  or  training  capabilities  of  the  test 
article  differed  from  the  intended  production  item  should  be  included.  DTE 
events  and  results  related  to  performance  characteristics,  critical  issues, 
requirements  levied  by  review  boards,  etc.,  should  be  emphasized.  Technical 
characteristics  and  specification  requirements  which  were  demonstrated  (or 
failed  to  be  demonstrated)  should  be  addressed.  The  results  and  decisions  of 
any  program  reviews  should  be  shown. 

(2)  Future  DTE.  This  section  addresses  all  remaining  DTE  commencing 
with  the  date  of  the  current  DTEMP  revision  and  extending  through  DT-IV.  Each 
remaining  phase  of  DTE  will  be  addressed  individually  and  include  the  four  sub- 
sections below. 


(a)  Equipment  Description.  This  description  emphasizes  the 
training  capability  and  how  it  is  expected  to  differ  from  the  model  previously 
tested  in  DTE/OTE  and  the  production  model. 

(b)  DTE  Objectives.  Include  the  specific  objective  of  each 
phase  or  subphase  of  future  DTE.  These  objectives  are  related  to,  but  probably 
not  the  same  as,  the  overall  program  objectives.  If  the  program  source  documents 
require  demonstration  of  particular  technical  characteristics  in  a given  DT 
phase,  these  characteristics  are  to  be  included. 

(c)  DTE  Events/Scope  of  Testing/Basic  Scenarios.  This  sub- 
section includes  all  T&E  events  which  will  provide  data  with  which  to  assess  the 
device  against  the  objectives.  The  scope  and  basic  test  scenarios  should  be 
described  in  sufficient  detail  so  that  the  relationship  between  the  tests  and 
the  objectives  is  apparent. 

(d)  Quantifiable  Scope  of  Effort.  This  is  a brief  summary  of 
the  key  elements  of  the  testing  expressed  in  terms  of  a measurable  output. 
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(3)  Critical  T&E  Items.  This  section  highlights  any  item  whose 
availability  is  critical  to  the  performance  of  adequate  DTE  prior  to  the  next 
decision  point.  Should  the  critical  item  not  be  available,  then  the  next  decision 
point  may  be  delayed.  Critical  items  may  be  displayed  on  the  Integrated  Schedule 
if  appropriate. 

e.  Part  V.  OTE  Outline.  The  OTE  Outline  is  prepared  by  the  OTA.  It 
addresses  all  OTE'  from  the  earliest  conceptual  phase  of  IOTE  through  the  final 
phase  of  FOTE.  The  OTE  Outline  will  show  all  OTE  in  sufficient  detail  that 
resources  can  be  identified,  and  the  OTA  can  develop  test  plans  from  it.  The 
sections  and  subsections  required  by  the  OTE  Outline  are  the  same  as  those  in 
the  DTE  Outline  (Part  IV). 

f.  Part  VI.  Resource  Summary.  This  part  contains  a combined  summary,  in 
tabular  form,  of  the  resources  required  for  all  T&E.  Listed  on  the  summary  form 
are  11  specific  resource  categories  likely  to  be  required.  Some  listed  categories 
may  not  be  required  for  a particular  program;  they  should  contain  a "not  applicable" 
statement.  Should  additional  categories  be  required,  they  should  be  listed. 

For  each  listed  category,  show  the  major  requirement  (what,  how  much,  how  many, 
etc.)  at  the  times  they  are  needed.  If  the  tabular  summary  does  not  allow 
adequate  space  to  define  essential  resource  requirements  in  sufficient  detail, 
an  additional  page  can  be  added.  Where  possible,  resources  should  be  shown  in 
kind  rather  than  in  dollar  terms.  The  summary  should  include  the  required 
location  of  each  resource  and,  where  applicable,  the  disposition  after  completion 
of  testing.  If  resources  are  already  committed  to  a program,  these  should  be 
included  and  listed.  Resource  requirements  for  DTE  should  be  listed  separately 
from  those  required  for  OTE.  Where  one  resource  can  be  used  for  both  DTE  and 
OTE,  this  should  be  indicated  on  the  summary.  As  an  aid  to  developing  the 
resource  requirements  in  each  of  the  categories  listed,  a brief  explanation  is 
presented. 


(1)  Test  Articles.  The  actual  number  of  test  articles  required  for 
each  major  type  of  T&E  must  be  identified.  If  subsystems  (components,  assemblies, 
or  subassemblies)  are  to  be  tested  individually,  each  subsystem  and  the  quantity 
required  are  to  be  identified.  Specifically,  identify  requirements  for  advanced 
development  models,  engineering  development  models,  preproduction  prototypes, 
special  preproduction  prototypes,  and  production  models.  If  a number  of  test 
systems  are  to  be  produced,  indicate  by  serial  number  when  each  system  is  required. 

(2)  RDT&E  Support.  RDT&E  Support  is  divided  into  two  categories-- 
that  provided  by  operational  fleet  units  (Fleet  RDT&E  Support)  and  that  provided 
by  a shore  establishment  command  (Other  RDT&E  Support). 

(a)  Fleet  RDT&E  Support.  The  number  of  ship-days,  aircraft- 
hours,  and  types  of  ships  and  aircraft  should  be  estimated.  If  support  is 
constrained  to  a specific  area  (Atlantic,  Pacific)  or  to  a specific  ship  or 
aircraft,  so  indicate.  Time  required  for  installation  and  removal  of  test 
systems  and  test-associated  equipment  should  be  indicated.  A distinction  should 
be  made  between  dedicated,  concurrent,  and  NIB  support  requirements.  Include  an 
estimate  of  the  number  of  personnel  who  will  be  aboard  each  ship  for  T&E  purposes, 
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not  including  ship's  company.  Fleet  KDT& E Support  required  soley  for  "target" 
purposes  should  be  identified  as  such.  (The  services  of  nonfleet-control ied 
resources  such  as  yard  tugs,  barges,  and  ancillary  equipment  are  entered  under 
the  Support  Equipment  Category.) 

(b)  Other  RDT&E  Support.  The  tv  nber  of  days  and  the  number  and 
type  of  facilities  required  to  perform  the  T&E  should  be  estimated.  If  support 
is  constrained  to  a specific  locale  or  to  a specific  facility  or  type  of  facility, 
so  indicate.  Time  required  for  the  installation  and  removal  of  test  device(s) 
and  test-associated  equipment  should  be  indicated.  A distinction  should  be  made 
between  dedicated,  concurrent,  and  NIB  support  requirements.  Include  an  estimate 
of  the  number  of  personnel  who  will  be  required  at  each  facility  for  T&E  purposes, 
not  including  permanently  assigned  personnel. 

(3)  Test  Sites/Ranges.  Test  sites  and  ranges  to  be  used  for  T&E,  and 
when  they  are  required,  are  to  be  listed.  Usage  time  is  to  be  estimated  in  days 
and  hours  per  day.  When  the  test  site  or  range  is  identified,  the  normal  instru- 
mentation of  that  facility  is  expected  to  be  available.  Resource  requirements 
for  modifying  existing  facilities  or  developing  new  facilities  will  be  included 
under  this  resource  heading. 

(4)  Special  Instrumentation.  Special  instrumentation  requirements 

for  T&E  should  be  identified  "(when  and  where  required).  Instrumentation  installed 
at  test  sites,  ranges,  or  facilities  which  will  be  available  under  normal  circum- 
stances need  not  be  identified  separately  from  the  site,  range,  or  facility. 

The  source  of  the  special  instrumentation,  the  time  required  for  instal lation, 
and  the  installing  activity  will  be  included. 

(5)  Support  Equipment.  Support  equipment  is  equipment  required  to 
conduct  a test,  but  is  not  a part  of  the  test  itself.  Support  equipment  should 
be  identified  by  type,  number  required,  date  required,  time  needed,  and  location. 
Support  equipment  which  has  standard  instal lation/ removal  factors  or  costs  need 
only  be  noted  as  requiring  installation/removal.  Installation  and  removal  time 
of  other  support  equipment  is  to  be  estimated  and  the  installing  and  removal 
activity  identified. 

(6)  Installation/Removal  Requirements.  Support  equipment  is  a 
separate  resource  requirement.  The  installation  and  removal  requirements  for 
equipments,  including  test  articles  which  are  actually  used  in  tests  to  be 
conducted,  will  be  summarized.  If  the  installation  and  removal  is  initially  for 
DTE,  and  the  same  equipment  will  be  used,  in  place,  for  OTE,  this  will  be 
indicated.  The  installing/removing  activity,  estimated  man-days  required,  and 
the  work  site  should  be  identified. 

(7)  Expendables.  Included  here  are  items  expended  during  tests,  not 
including  test  items  or  targets.  Also  included  are  specialized  supplies  not 
normally  used,  or  used  in  excess  of  the  normal  supply,  by  the  test  activity, 
test  site,  or  supporting  unit(s).  Include  number  and  type  required  and  the 
date  and  location  for  the  expenditure. 
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(8)  Logistic  Support.  Requirements  for  repair  parts,  spares,  etc., 
in  excess  of  the  normal  support  package  provided  with  the  test  article(s)  is  to 
be  shown.  Include  extra  spares  nece  sary  to  support  other  equipments  used  in 
conjunction  with  the  test. 

(9)  Personnel . Estimate  personnel  requirements  in  man-days  per 
calendar  period.  Rank/rate/grade,  number  of  personnel,  and  when  and  where  they 
are  required  are  entered.  Analytic  and  simulation  support  personnel  should  be 
tabulated  separately  from  test  personnel.  Requirements  for  personnel  other  than 
test,  analytic,  and  simulation  support  should  be  identified  and  entered  as  a 
distinct  category.  If  contractors  are  hired  solely  for  testing,  analytical,  or 
simulation  support,  the  man-weeks  of  contract  support  should  be  estimated. 


(10)  Personnel  Training.  All  test  personnel  and  fleet  or  other  source  t 

personnel  who  require  training  for  the  testing,  including  operators  and  maintenance 
personnel,  are  to  be  included.  Training  of  DA  and  OTA  test  supervisors  and 
observers  must  also  be  included.  Identify  the  rate/rank,  number  of  personnel,  • 

source,  and  when  the  training  should  be  complete. 


(11)  Planned  Travel . This  entry  is  required  to  permit  long-range 
budgeting  for  travel  and  per  diem.  Estimate  planned  travel  in  dollar  terms  by 
FY,  subdivided  by  the  command  responsible  for  funding. 

(12)  Other.  Include  other  categories  of  resources  required  as  necessary. 

g.  Part  VII.  References.  This  part  should  list  pertinent  reports 
containing  results  of  accomplished  test  and  evaluation.  In  addition,  developed 
test  plans  can  be  referenced  for  more  detailed  information. 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  NO. 

(Program  Short  Title) 

Part  I 

Administrative  Information 

Full  Program  Title 

DCP/PM/NDCP  No. 

Program  Element  No. Project  No.  

S&TO  No. OR  No. 

DP  No. 

AC  AT DCAT DA 

Points  of  Contact 

Title Name/Code Agency  Ph.No. 

Assessment  Sponsor 
Resource  Sponsor 
Program  Coordinator 
Development  Coordinator 
PM/AM 

CHNAVMAT  Coordinator 
CNET  Coordinator 
DA  Test  Director 
Operational  Test  Director 
(Others  as  required) 
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5.  Funding  and  Procurement 


Part  I 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  NO. 
(Program  Short  Title) 

6.  Delivery  and  Installation 

a.  Delivery  Schedule  (Example): 


(Date  Revised) 


FISCAL-YEAR 

]9XX  19XX  19XX  19XX.  19XX  19XX  19XX 
Preproduction  Prototypes  1 

Special  Pilot  Production  1 

Production  1 

b.  Programmed  Installation  (Example): 

FY  19XX  - DT  I I/AT  SSC,  Great  Lakes. 

FY  19XX  - TECHEVAL/TRAEVAL  AT  SSC,  Great  Lakes 
FY  19XX  - OT-V  at  SSC,  San  Diego. 

7.  Remarks 
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a.  Operational  System  Description 

b.  Functional  (Training  Mission)  Description 

c.  Hardware  Description 

Critical  T&E  Issues  (From  Document  No.  of 


Objectives  and  Thresholds 

a.  Program  Objectives 

(1) 

(2) 

b.  Program  Thresholds 

(1)  Cost 
(a) 

(2)  Performance 
(a) 

(3)  Schedule 
(a) 

Required  Technical  Characteristics  (From 


(date) ) 


source)  of 
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Required  Operational  Characteristics  (From 


source! 


Training  Effectiveness 


b.  Operational  Suitability 

(1)  Reliability 

(2)  Maintainability 

(3)  Availability 

(4)  Logistic  Supportabil ity 

(5)  Compatibility 

(6)  Interoperability 

(7)  Training  (user) 

(8)  Transportability 

(9)  Human  Factors 

(10)  Safety 

NOTE:  Certain  aspects  of  specific  required  operational  characteristics  may  be 
expressed  in  quantitative  terms,  others  will  require  a qualitative 
expression. 

6.  Environmental  Impact  Assessment  of  T&E 

a.  Probable  impact  of  required  T&E  on  the  environment 

b.  Alternatives 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  NO. 
(Program  Short  Title) 


DATE  REVISED 


Legend 

(As  Required) 
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OPNAVINST 
(Date  Revised) 

DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  NO. 

(Program  Short  Title) 

PART  IV 
DTE  OUTLINE 

1 . DTE  to  Date 

a.  Applicable  DTE  Phase(s)  (DT-I,  DT-II,  etc.) 

(1)  Equipment  Description 

(2)  DTE  Events  and  Results 

(3)  Program  Management  Reviews/Decisions 

2.  Future  DTE 

A.  Applicable  DTE  Phase(s)  and  Inclusive  Dates  (DT-I 1 1 A,  DT-III  B,  etc.) 

(1)  Equipment  Description 

(2)  DTE  Objectives 

(3)  DTE  Events/Scope  of  Testing/Basic  Scenarios 

(4)  Quantifiable  Scope  of  Effort 

3.  Critical  Items 

a. 

b. 

NOTE:  Each  DT  Phase  will  be  discussed  in  an  independent  subparagraph. 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN  NO. 

(Program  Short  Title) 

PART  V 
OTE  OUTLINE 

OTE  to  Date 

a.  Applicable  OTE  Phase(s)  (OT-I,  OT-II,  etc.) 

(1)  Equipment  Description 

(2)  OTE  Events  and  Results 

(3)  Program  Management  Reviews/Decisions 
Future  OTE 

a.  Applicable  OTE  Phase(s)  and  Inclusive  Dates  (OT-II I A,  OT-II I B,  etc.) 

(1)  Equipment  Description 

(2)  OTE  Objectives 

(3)  OTE  Events/Scope  of  Testing/Basic”  Scenarios 

(4)  Quantifiable  Scope  of  Effort 
Critical  Items 

a. 

b. 

Each  OT  Phase  will  be  discussed  in  an  independent  paragraph. 
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PART  VI  - RESOURCE  SUMMARY 

(Include  Only  Rows  Actually  Containing  Entries) 


RESOURCE 


DEVICE  TEST  AND  EVLUATION  MASTER  PLAN  NC 
(Program  Short  Title) 

DATE  REVISED 


Test  Articles 


RDT&E 

Support 


Fleet 


Test  Site/Ranges 


Special 

Instrumentation 


Support 

Equipment 


Installation 
Removal  Requirement 


Other 


OPNAVINST 


EVICE  TEST  AND  EVLUATION  MASTER  PLAN  NO. 
(Program  Short  Title) 


TAEG  Report  No.  71 


OPNAV INST 


REQUESTS  FOR  RDT&E  SUPPORT 


1.  Fleet  RDT&E  Support 

a.  Requests  for  Fleet  RDT&E  Support  are  used  only  to  obtain  support  for 
research  and  development  not  related  to  specific  acquisition  programs.  All 
other  Fleet  RDT&E  Support  is  obtained  through  TEMPs  or  DTEMPs. 

b.  Requests  for  Fleet  RDT&E  Support  to  be  used  in  the  development  of 
training  devices  are  submitted  by  the  cognizant  R&D  Agency,  via  CNET,  to  CNO 
(OP-983)  for  approval.  Copies  of  each  request  will  be  distributed  to  CINCPACFLT, 
CINCLANTFLT,  CHNAVMAT,  COMOPTEVFOR,  DEPCOMOPTEVFORPAC , appropriate  type  commanders, 
and  may  be  sent  to  other  interested  commands.  COMNAVSEASYSCOM  will  be  included 

in  the  distribution  when  ship  weight  and  moment  or  safety  considerations  are 
involved.  COMNAVAIRSYSCOM  will  be  included  in  the  distribution  when  aircraft 
installations  are  involved. 

c.  Every  effort  must  be  made  to  keep  requested  services  to  an  absolute 
minimum. 

d.  Requests  will  be  submitted  at  least  3 months  before  the  start  of  the 
earliest  quarter  in  which  support  is  required.  For  continuing  programs,  each 
request  will  contain  both  short-term  and  long-term  requirements.  Short-term 
requirements  are  those  in  the  18-month  period  following  submission  of  the 
request.  These  must  be  stated  in  sufficient  detail  to  permit  their  use  in 
preparing  the  CNO  Quarterly  Fleet  RDT&E  Support  Requirements  for  Fleet  Scheduling. 
Long-term  requirements  are  those  from  the  end  of  the  short-term  period  to  the 

end  of  the  research  or  development  program  (but  no  more  than  5 years).  Reason- 
able care  must  be  taken  in  estimating  long-term  requirements  because  these 
estimates  are  used  in  planning,  programming,  and  budgeting  for  Fleet  RDT&E 
Support. 


e.  Requests  will  be  updated  and  resubmitted  when  significant  changes 
occur,  and  at  least  annually. 

f.  Tab  A is  the  format  for  Fleet  RDT&E  Support  requests. 

g.  One-time  requests  for  support  may  be  submitted  by  message. 

h.  If  requests  are  approved  by  CNO,  OP-098  will  assign  a T&E  identifi- 
cation number  (and  priority,  for  imminent  projects),  and  direct  CNET  to  schedule 
the  Fleet  RDT&E  Support. 

2.  Other  RDT&E  Support. 

a.  Requests  for  Other  RDT&E  Support  are  used  only  to  obtain  support  for 
research  and  development  not  related  to  specific  training  device  acquisition 
programs.  All  Other  RDT&E  Support  related  to  a specific  acquisition  program  is 
obtained  through  DTEMPs. 
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b.  Requests  for  Other  RDTAE  Support  will  be  confined  to  programs  directly 
related  to  training  device  development. 

c.  Requests  for  Other  RDTAE  Support  to  be  used  in  the  development  of 
training  devices  are  submitted  by  the  cognizant  RAD  Agency,  via  CNET,  to  CNO 
(OP-983)  for  approval.  Copies  of  each  request  will  be  distributed  to  the 
immediate  comnander  of  the  facility  being  requested,  the  facility  being  re- 
quested, and  may  be  sent  to  other  interested  commands. 

d.  Every  effort  must  be  made  to  keep  requested  services  to  an  absolute 
minimum. 

e.  Requests  will  be  submitted  at  least  3 months  before  the  start  of  the 
earliest  quarter  in  which  support  is  required,  lor  continuing  programs,  each 
request  will  contain  both  short-term  and  long-term  requirements.  Short-term 
requirements  are  those  in  the  18-month  period  following  submission  of  the 
request.  These  must  be  stated  in  sufficient  detail  to  permit  their  use  in 
scheduling  the  support.  Long-term  requirements  are  those  from  the  end  of  the 
short-term  period  to  the  end  of  the  research  or  development  program  (but  no  more 
than  S years).  Care  must  be  taken  in  estimating  long-term  requirements  because 
these  estimates  are  used  to  plan,  program,  and  budget. 

f.  Requests  will  be  updated  and  resubmitted  when  significant  changes 
occur,  and  at  least  annually. 

g.  Tab  B is  the  format  for  Other  RDTAE  Support  requests. 

h.  One-time  requests  for  support  may  be  submitted  by  message. 

i.  If  requests  are  approved  by  CNO,  OP-983  will  assign  a TAE  identifi- 
cation number,  a priority  for  imminent  projects,  and  direct  CNET  to  schedule  the 
support. 
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FORMAT  FOR  FLEET  RDT&E  SUPPORT  REQUESTS 


I . REQUIREMENTS 

1.  Specify  the  support  required,  including  type,  extent,  and  duration. 
State  requirements  in  terms  of  ship,  aircraft,  or  unit  type,  unless  a specific 
ship/aircraft/unit  is  required.  If  a specific  ship/aircraft/unit  is  required, 
state  why.  Include  any  unusual  training  or  operating  requirements  the  support- 
ing unit(s)  must  meet. 

a.  Short-term  requirements.  State  the  planned  start-date  for  each 
discrete  period  of  support  requested. 

b.  Long-term  requirements . State  the  FY  quarter  in  which  each 
discrete  period  of  support  will  be  required. 

I I . PURPOSE 

1.  Identify  the  program  for  which  support  is  requested,  including  program 
element  number  and  project  number. 

2.  Briefly  state  the  purpose  of  the  program. 

3.  For  each  short-term  requirement,  specify: 

a.  Planned  tests,  indicating  in  general  how  the  tests  will  be 

performed. 

b.  Special  conditions  needed  for  the  tests,  such  as  weather,  geo- 
graphic area,  geometry  of  tests,  etc. 

c.  When  the  test  plan  will  be  available. 

III.  DESCRIPTION  AND  STATUS  OF  EQUIPMENT  (Short-term  support  requirements  only) 

1.  Briefly  describe  system/equipment  to  be  installed  or  taken  aboard. 
Include  power,  weight  and  moment  considerations,  possible  effects  on  ship/ 
aircraft  performance,  etc. 

2.  Estimate  number  and  qualifications  of  personnel  needed  for  operation 
and  maintenance  of  the  equipment  or  system. 

3.  State  whether  shipyard  or  tender  availability  will  be  required  for 
installation,  including  any  special  test  or  data  collection  equipment. 

4.  State  estimate  of  installation/removal  time. 

5.  State  whether  funds  for  installation/removal  are  or  will  be  available. 
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6.  State  whether  installation  plan  has  been  forwarded,  or  date  it  will  be 
forwarded,  to  installing  activity. 

7.  Provide  disposition  instructions  for  material  on  completion  of  the 

vCj [ J • 

8 Where  applicable,  identify  approving  authority  for  weight  and  moment 
or  safety  considerations. 


IV.  CLASSIFICATION  AND  SECURITY  (All  requests) 


1;.  Specify  security  measures  (physical,  operational,  electronic),  ranqe 
precautions,  information  not  releasable  to  foreign  nationals,  or  any  special 
considerations  such  as  patent  rights,  foreign-origin  equipment,  etc. 


2.  Specify  classification  of  the  subject  of  the  program,  if  overall 
project  is  classified. 


3.  Specify  classification  of  equipment  capabilities  and  performance. 

V.  REMARKS 

,In^1cate  availability  of  technical  personnel  from  requesting  activity 
for  technical  support.  J 

2.  Indicate  number  of  personnel  to  embark  in  fleet  unit(s).  (Short-term 
requirements  only.) 

. 3-  List  names  of  individuals  who  are  immediately  responsible  for  the 
project  and  who  can  act  in  a liaison  capacity.  Include  organization,  organiza- 
tion code,  and  telephone  number. 
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FORMAT  FOR  OTHER  RDT&E  SUPPORT  REQUESTS 


I.  REQUIREMENTS 

1.  Specify  the  support  required,  including  type,  extent,  and  duration. 
State  requirements  in  terms  of  activity  type,  unless  a specific  activity  is 
needed.  If  a specific  activity  is  needed,  state  why.  Include  any  unusual 
training  or  other  requirements  the  supporting  activity(ies)  must  meet. 

a.  Short-term  requirements.  State  the  planned  start-date  for  each 
discrete  period  of  support  requested. 

b.  Long-term  requirements.  State  the  FY  quarter  in  which  each 
discrete  period  of  support  will  be  required. 

II.  PURPOSE 

1.  Identify  the  program  for  which  support  is  requested,  including  program 
element  number  and  project  number. 

2.  Briefly  state  the  purpose  of  the  program. 

3.  For  each  short-term  requirement,  specify: 

a.  Planned  tests,  indicating  in  general  how  the  tests  will  be 

performed . 


b.  Special  conditions  needed  for  the  tests,  such  as  weather,  geo- 
graphic area,  geometry  of  tests,  etc. 

c.  When  the  test  plan  will  be  available. 

III.  DESCRIPTION  AND  STATUS  OF  EQUIPMENT  (Short-term  support  requirements  only) 

1.  Briefly  describe  system/equipment  to  be  installed. 

2.  Estimate  number  and  qualifications  of  personnel  needed  for  operation 
and  maintenance  of  the  equipment  or  system. 

3.  State  who  will  install  and  remove  equipment. 

4.  State  installation/removal  time. 

5.  State  whether  funds  for  installation/removal  are  or  will  be  available. 

6.  State  whether  installation  plan  has  been  forwarded,  or  date  it  will  be 
forwarded,  to  installing  activity. 

7.  Provide  disposition  instructions  for  material  on  completion  of  the 
tests. 
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IV.  CLASSIFICATION  AND  SECURITY  (All  requests) 

1.  Specify  security  measures  (physical,  operational,  electronic),  range 
precautions,  information  not  releasable  to  foreign  nationals,  or  any  special 
considerations  such  as  patent  rights,  foreign-origin  equipment,  etc. 

2.  Specify  classification  of  the  subject  of  the  program,  if  overall 
project  is  classified. 

3.  Specify  classification  of  equipment  capabilities  and  performance. 

V.  REMARKS 

1.  Indicate  availability  of  technical  personnel  from  requesting  activity 

for  technical  support.  J 

2.  Indicate  number  of  personnel  to  be  billeted  at  the  activity  providing 
support  over  and  above  the  authorized  allowance.  (Short-term  requirements 

3.  List  names  of  individuals  who  are  immediately  responsible  for  the 

project  and  who  can  act  in  a liaison  capacity.  Include  organization,  organiza- 
tion code,  and  telephone  number.  3 
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REPORT 

REPORT  SYMBOLS 

SUBMITTED  BY 

OPNAV  REPORT  SYMBOL 

CNO  Index  of  Acquisition 
Programs 

OP -098 

3960-5 

*Test  and  Evaluation 

DA  and  COM- 

3960-6 

Master  Plan  (TEMP) 

OPTEVFOR 

^Device  Test  and  Evaluation 
Master  Plan  (OTEMP) 

DA  and  OTA 

Request  for  Fleet  RDT&E 

Support  (for  nonacquisition 
programs ) 

R&D  Agency 

3960- 7A 

Request  for  Other  RDT&E  Support 
(for  nonacquisition  programs) 

R&D  Agency 

CNO  Long-Range  Fleet  RDT&E 
Support  Requirements 

OP-098 

3960- 7B 

CNO  Long-Range  Other  RDT&E 
Support  Requirements 

OP-098 

CNO  Quarterly  Fleet  RDT&E 
Support  Requirements 

OP-098 

3960- 7C 

Quarterly  Report  of  Fleet 

RDT&E  Support  Provided 

C0M0PTEVF0R/0TA 

3960-7D 

Quarterly  Report  of  Other 

RDT&E  Support  Provided 

OTA  or  DA 

*DT&E  Report 

DA 

3960-8** 

Certification  of  Readiness 
for  OPEVAL 

DA 

3960-9 

Certification  of  Readiness  for 
TRAEVAL 

DA 

Ship  Readiness  for  OPEVAL 

Ship  CO 

3960-10 

Deficiency  Report 

C0M0PTEVF0R/0TA 

3960-11 

*0T&E  Report 

COMOPTEVFOR/OTA 

3960-12 

*COMOPTEVFOR  Tactics 

Guide 

COMOPTEVFOR 

3960-13 
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REPORT  SYMBOLS  (Continued) 

♦These  reports  will  carry  the  following  notation: 

"Distribution  limited  to  U.S.  Government  agencies  only;  Test  and 
Evaluation;  (date  of  report).  Other  requests  for  this  document 
must  be  referred  to  the  Chief  of  Naval  Operations  (OP-098)." 

♦♦Many  types  of  DT&E  Reports  are  assigned  other  symbols,  and  should  continue 
with  them.  3960-8  is  assigned  for  major  DT&E  reports  (e.g.,  TECHEVAL  reports), 
or  for  DT&E  reports  not  carrying  other  symbols. 
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Acceptance  Tests  Tests  conducted  by  or  for  the  Developing  Agent 

prior  to  government  acceptance  of  the  training 
device.  The  objective  of  these  tests  is  to 
insure  the  device  meets  the  government’s  techni- 
cal specifications. 


Acquisition  Categories 
(ACAT) 


Approval  for  Service 
Use  (ASU) 


Device  Test  and  Evaluation 
Master  Plan  (DTEMP) 


An  assigned  category  based  on  nominal  value 
thresholds  or  other  criteria  which  define  the 
level  of  attention  in  the  acquisition  process 
and  the  degree  of  T&E  required. 

An  approval  of  a training  device  prior  to  the 
production  decision  to  insure  the  device  meets 
operational  and  technical  requirements  within 
the  training  system  for  both  performance  and 
supportability,  and  is  training  effective. 

Identical  to  the  TEMP,  except  applicable  only  to 
training  devices. 


Design  Freeze  That  point  in  the  developmental  effort  at  which 

the  training  device  design  is  agreed  upon  by  both 
the  government  and  the  vendor.  Any  subsequent 
change  to  the  design  requires  a contract  modifi- 
cation. 


Development  Agent  (DA)  The  Systems  Command  or  Service  Agency  assigned 

responsibil ity  for  the  development  and  develop- 
ment test  and  evaluation  of  a training  device, 
subsystem  of  the  device,  or  an  item  of  training 
equipment. 


Direct  Training  Device  A requirement  established  to  support  a training 

Requirement  system  for  a new  or  modified  operational  system 

or  hardware. 


Fleet  Project  Team  (FPT) 


Hardware  Acquisition 
Manager 


A group  of  knowledgeable  representatives  from 
the  fleet  or  other  user  and  interested  nonuser 
activities,  consisting  of  qualified  military 
and/or  civilian  personnel  designated  by  cognizant 
commands.  The  FPT  will  assist  and  advise  the 
training  device  development  and  acquisition 
activity  in  development,  acquisition,  and  accept- 
ance of  specifically  assigned  training  devices. 

The  activity  or  command  having  direct  technical 
contractual  responsibility  and  authority  to 
develop  and/or  acquire  a specific  operational 
system  or  equipment. 
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GLOSSARY  Or  TERMS  USED  IN  THIS  REPORT  (continued) 

Independent  Test  That  operational  test  and  evaluation  conducted  by 

and  Evaluation  a field  agency  which  is  separate  and  distinct 

from  the  developing/procuring  command  and  from 
the  using  command. 


Indirect  Training  Device 
Requirement 


A training  device  requirement  identified  within 
the  training  community  to  support  an  existing  or 
proposed  course  of  instruction. 


Job  Performance 
Measures  (JPM) 

Operational  Test 
Agent  (OTA) 

Operational  Test 
and  Evaluation  (OTE) 


Prototype  Training 
Device 


Service  Acceptance 


Test  and  Evaluation 
Master  Plan  (TEMP) 


Training  Device 
Acquisition  Activity 


Tests  that  are  used  to  evaluate  the  proficiency 
of  a job  holder  on  each  task  he  performs. 

The  Navy  agent  designated  to  conduct  an  indepen- 
dent operational  test  and  evaluation. 

That  training  device  test  and  evaluation  conducted 
to  estimate  the  prospective  training  effectiveness 
and  operational  suitability,  and  the  need  for  any 
modifications  to  the  device  or  training  system 
of  which  it  is  a part. 

The  first  unit  or  units  acquired  with  RDT&E  funds. 
This  is  the  preproduction  unit(s)  acquired  to 
insure  the  device  meets  operational  and  development 
requirements. 

Acceptance  by  the  Navy  of  the  training  device  from 
the  vendor  for  inclusion  in  the  Navy  inventory. 

The  overall  test  and  evaluation  plan  to  identify 
and  integrate  the  effort  and  schedules  of  all  T&E 
to  be  accomplished  and  to  insure  that  all  necessary 
T&E  is  accomplished  prior  to  the  key  decision 
point. 

The  activity  or  command  having  direct  technical 
and  contractual  responsibility  and  authority 
to  develop  and/or  acquire  a specific  training 
device. 
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DISTRIBUTION  LIST 

Air  Force 

Headquarters,  Air  Training  Command  (XPTD) 

Headquarters,  Air  Training  Command  (XPTIA,  Mr.  Goldman) 

Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  Base 
Air  Force  Human  Resources  Laboratory  (Library),  Lowry  Air  Force  Base 
Air  Force  Office  of  Scientific  Research/AR  (Dr.  A.  R.  Fregly) 
Headquarters  Tactical  Air  Command  (DOOS)  Langley  Air  Force  Base 


Army 

Commandant,  TRADOC  (Technical  Library) 

ARI  (Dr.  Ralph  R.  Canter,  316C;  Dr.  Edgar  Johnson;  Mr.  James  Baker) 
Dr.  H.  F.  O'Neil,  Jr.,  Dr.  Beatrice  Farr,  PERI-OK) 


Coast  Guard 

Commandant,  U.S.  Coast  Guard  Headquarters  (G-P-l/62;  G-RT/81) 


Marine  Corps 
CMC  (OT) 

CGMCDEC  (Mr.  Greenup) 

Director,  Marine  Corps  Institute 
CO  MARCORCOMMELECSCOL  (ITS)  (Mr.  D.  E.  Mueller) 


Navy 

OASN  (R&D)  (Dr.  S.  Koslov)  (MRA&L)  (Dr.  B.  D.  Rostker) 

CNO  (0P-102X,  M.  Malehorn;  0P-987P10,  Dr.  R.  Smith;  OP-987,  Mr.  H.  Stone;  0P-02, 
0P-03,  0P-04,  0P-05,  OP-096,  OP-983) 

NAVCOMPT  (NCD-7) 

ONR  (458,  455) 

CNM  (MAT-08T2,  Mr.  A.  L.  Rubinstein;  MAT-04) 

CNET  (01,  00A,  N-5  (6  copies),  015  (5  copies),  N-2,  N-3,  N-34) 

CNAVRES  (02) 

COMNAVSEASYSCOM  (03.05L1C,  05L1C2) 

COMNAVAIRSYSCOM  (03,  340F) 

CNET  SUPPORT  (00,  00A  (2  copies)) 

CNTECHTRA  (016,  Dr.  Kerr  (5  copies)) 

CNATRA  (Dr.  F.  Schuf letowski ) 

COMTRALANT 

COMTRALANT  (Educational  Advisor) 

COMOPTEVFOR 


Page  1 of  2 


DISTRIBUTION  LIST  (continued) 


Navy  (continued) 

COMTRAPAC  (2  copies) 

CO  NAVPERSRANDCEN  (Library  (4  copies)) 

NAVPERSRANDCEN  Liaison  (01H) 

Superintendent  NAVPGSCOL  (2124) 

Superintendent  Naval  Academy  Annapolis  (Chairman,  Behavioral  Science  Dept.) 
CO  NAVEDTRASUPPCEN  NORVA 
CO  NAVEDTRASUPPCENPAC  (5  copies) 

CO  NAMTRAGRU 

CO  NAVTRAEQUIPCEN  (N-424  (2  copies)),  N-211,  N-001 , N-002,  PM) 

Center  for  Naval  Analyses  (2  copies) 

OIC  NODAC  (2) 

CO  NAVSUBTRACENPAC  (2  copies) 

CO  FLETRACEN  SDIEGO 
CISO,  SSC  (Dr.  Flesch) 

Executive  Director  NAVINSTPRODEVDET 

CO  NAVTECHTRACEN  Corry  Station  (Cryptologic  Training  Department) 

NETSCPAC  Field  Office  Mare  Island 

Supply  Schools  Training  Officer  (Code  730),  Meridian,  MS 


Other  DOD 


Military  Assistant  for  Human  Resources,  OUSDR&E,  Room  3D129,  Pentagon 
(CDR  Paul  Chatelier) 

OASD  (MRA&L)  (LT  COL  Grossel ) 

Director,  Human  Resources  Office,  Defense  Advanced  Research  Projects  Agency 
(Dr.  Dexter  Fletcher) 

Institute  for  Defense  Analyses  (Dr.  Jesse  Orlansky) 


Information  Exchanges 

DDC  (12  copies) 

DLSIE  (Mr.  James  Dowling) 

ERIC  Processing  and  Reference  Facility,  Bethesda,  MD  (2  copies) 
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